{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Tutorial for <font color='green'> Toad</font>\n",
    "\n",
    "\n",
    "Toad is developed to facilitate the model development of credit risk scorecard particularly. In this tutorial, the basic use of toad will be introduced. \n",
    "\n",
    "______\n",
    "______\n",
    "\n",
    "The tutorial will follow the common procedure of credit risk scorecard model development: \n",
    "\n",
    "\n",
    "\n",
    "**(1) EDA**\n",
    "\n",
    "**(2) Feature selection and WOE binning**\n",
    "\n",
    "**(3) Model selection**\n",
    "\n",
    "**(4) Model validation**\n",
    "\n",
    "**(5) Scorecard transformation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0. Data preparation\n",
    "The data used is the famous german credit dataset. The following data preprocessing include: \n",
    "\n",
    "(1) replacing target as 0, 1 from 'good' / 'bad';\n",
    "\n",
    "(2) train test split; \n",
    "\n",
    "and (3) adding a feature to indicate train and test. The training set will be used for modelling and the test set will only be used for validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "! Make sure to upgrade to the latest version !\n",
    "'''\n",
    "#!pip install --upgrade toad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import toad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 21)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>status.of.existing.checking.account</th>\n",
       "      <th>duration.in.month</th>\n",
       "      <th>credit.history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit.amount</th>\n",
       "      <th>savings.account.and.bonds</th>\n",
       "      <th>present.employment.since</th>\n",
       "      <th>installment.rate.in.percentage.of.disposable.income</th>\n",
       "      <th>personal.status.and.sex</th>\n",
       "      <th>other.debtors.or.guarantors</th>\n",
       "      <th>...</th>\n",
       "      <th>property</th>\n",
       "      <th>age.in.years</th>\n",
       "      <th>other.installment.plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>number.of.existing.credits.at.this.bank</th>\n",
       "      <th>job</th>\n",
       "      <th>number.of.people.being.liable.to.provide.maintenance.for</th>\n",
       "      <th>telephone</th>\n",
       "      <th>foreign.worker</th>\n",
       "      <th>creditability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>... &lt; 0 DM</td>\n",
       "      <td>6</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>radio/television</td>\n",
       "      <td>1169</td>\n",
       "      <td>unknown/ no savings account</td>\n",
       "      <td>... &gt;= 7 years</td>\n",
       "      <td>4</td>\n",
       "      <td>male : single</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>real estate</td>\n",
       "      <td>67</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>1</td>\n",
       "      <td>yes, registered under the customers name</td>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0 &lt;= ... &lt; 200 DM</td>\n",
       "      <td>48</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>radio/television</td>\n",
       "      <td>5951</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years</td>\n",
       "      <td>2</td>\n",
       "      <td>female : divorced/separated/married</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>real estate</td>\n",
       "      <td>22</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>no checking account</td>\n",
       "      <td>12</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>education</td>\n",
       "      <td>2096</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>male : single</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>real estate</td>\n",
       "      <td>49</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>unskilled - resident</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>... &lt; 0 DM</td>\n",
       "      <td>42</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>furniture/equipment</td>\n",
       "      <td>7882</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>male : single</td>\n",
       "      <td>guarantor</td>\n",
       "      <td>...</td>\n",
       "      <td>building society savings agreement/ life insur...</td>\n",
       "      <td>45</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>... &lt; 0 DM</td>\n",
       "      <td>24</td>\n",
       "      <td>delay in paying off in the past</td>\n",
       "      <td>car (new)</td>\n",
       "      <td>4870</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years</td>\n",
       "      <td>3</td>\n",
       "      <td>male : single</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>unknown / no property</td>\n",
       "      <td>53</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  status.of.existing.checking.account  duration.in.month  \\\n",
       "0                          ... < 0 DM                  6   \n",
       "1                   0 <= ... < 200 DM                 48   \n",
       "2                 no checking account                 12   \n",
       "3                          ... < 0 DM                 42   \n",
       "4                          ... < 0 DM                 24   \n",
       "\n",
       "                                      credit.history              purpose  \\\n",
       "0  critical account/ other credits existing (not ...     radio/television   \n",
       "1           existing credits paid back duly till now     radio/television   \n",
       "2  critical account/ other credits existing (not ...            education   \n",
       "3           existing credits paid back duly till now  furniture/equipment   \n",
       "4                    delay in paying off in the past            car (new)   \n",
       "\n",
       "   credit.amount    savings.account.and.bonds present.employment.since  \\\n",
       "0           1169  unknown/ no savings account           ... >= 7 years   \n",
       "1           5951                 ... < 100 DM       1 <= ... < 4 years   \n",
       "2           2096                 ... < 100 DM       4 <= ... < 7 years   \n",
       "3           7882                 ... < 100 DM       4 <= ... < 7 years   \n",
       "4           4870                 ... < 100 DM       1 <= ... < 4 years   \n",
       "\n",
       "   installment.rate.in.percentage.of.disposable.income  \\\n",
       "0                                                  4     \n",
       "1                                                  2     \n",
       "2                                                  2     \n",
       "3                                                  2     \n",
       "4                                                  3     \n",
       "\n",
       "               personal.status.and.sex other.debtors.or.guarantors  ...  \\\n",
       "0                        male : single                        none  ...   \n",
       "1  female : divorced/separated/married                        none  ...   \n",
       "2                        male : single                        none  ...   \n",
       "3                        male : single                   guarantor  ...   \n",
       "4                        male : single                        none  ...   \n",
       "\n",
       "                                            property age.in.years  \\\n",
       "0                                        real estate           67   \n",
       "1                                        real estate           22   \n",
       "2                                        real estate           49   \n",
       "3  building society savings agreement/ life insur...           45   \n",
       "4                              unknown / no property           53   \n",
       "\n",
       "   other.installment.plans   housing number.of.existing.credits.at.this.bank  \\\n",
       "0                     none       own                                       2   \n",
       "1                     none       own                                       1   \n",
       "2                     none       own                                       1   \n",
       "3                     none  for free                                       1   \n",
       "4                     none  for free                                       2   \n",
       "\n",
       "                           job  \\\n",
       "0  skilled employee / official   \n",
       "1  skilled employee / official   \n",
       "2         unskilled - resident   \n",
       "3  skilled employee / official   \n",
       "4  skilled employee / official   \n",
       "\n",
       "  number.of.people.being.liable.to.provide.maintenance.for  \\\n",
       "0                                                  1         \n",
       "1                                                  1         \n",
       "2                                                  2         \n",
       "3                                                  2         \n",
       "4                                                  2         \n",
       "\n",
       "                                  telephone foreign.worker creditability  \n",
       "0  yes, registered under the customers name            yes             0  \n",
       "1                                      none            yes             1  \n",
       "2                                      none            yes             0  \n",
       "3                                      none            yes             0  \n",
       "4                                      none            yes             1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('germancredit.csv')\n",
    "data.replace({'good':0,'bad':1},inplace=True) # replace target as 0, 1. \n",
    "\n",
    "print(data.shape) # 1000 data and 20 features \n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(750, 22)\n"
     ]
    }
   ],
   "source": [
    "Xtr,Xts,Ytr,Yts = train_test_split(data.drop('creditability',axis=1),data['creditability'],test_size=0.25,random_state=450)\n",
    "data_tr = pd.concat([Xtr,Ytr],axis=1)\n",
    "data_tr['type'] = 'train' # A new feature to indicate test/training sample\n",
    "data_ts = pd.concat([Xts,Yts],axis=1)\n",
    "data_ts['type'] = 'test'\n",
    "\n",
    "# The training set will be used for modelling and the test set will only be used for validation. \n",
    "\n",
    "print(data_tr.shape)\n",
    "#print(data_tr.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "----\n",
    "### I. EDA data handling\n",
    "Toad supports general EDA of each feature to detect missing values and feature distribution.\n",
    "\n",
    "1. **toad.detector.detect()**: return the EDA report of each feature, incl. data type, distribution, missing rate, and unique values. The EDA report is aimed to guide missing / extreme value detection & replacing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>size</th>\n",
       "      <th>missing</th>\n",
       "      <th>unique</th>\n",
       "      <th>mean_or_top1</th>\n",
       "      <th>std_or_top2</th>\n",
       "      <th>min_or_top3</th>\n",
       "      <th>1%_or_top4</th>\n",
       "      <th>10%_or_top5</th>\n",
       "      <th>50%_or_bottom5</th>\n",
       "      <th>75%_or_bottom4</th>\n",
       "      <th>90%_or_bottom3</th>\n",
       "      <th>99%_or_bottom2</th>\n",
       "      <th>max_or_bottom1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>status.of.existing.checking.account</th>\n",
       "      <td>object</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>4</td>\n",
       "      <td>no checking account:39.20%</td>\n",
       "      <td>... &lt; 0 DM:27.60%</td>\n",
       "      <td>0 &lt;= ... &lt; 200 DM:27.07%</td>\n",
       "      <td>... &gt;= 200 DM / salary assignments for at leas...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>no checking account:39.20%</td>\n",
       "      <td>... &lt; 0 DM:27.60%</td>\n",
       "      <td>0 &lt;= ... &lt; 200 DM:27.07%</td>\n",
       "      <td>... &gt;= 200 DM / salary assignments for at leas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>duration.in.month</th>\n",
       "      <td>int64</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>32</td>\n",
       "      <td>20.548</td>\n",
       "      <td>11.941</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>18</td>\n",
       "      <td>24</td>\n",
       "      <td>36</td>\n",
       "      <td>60</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>credit.history</th>\n",
       "      <td>object</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>5</td>\n",
       "      <td>existing credits paid back duly till now:53.73%</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>delay in paying off in the past:8.00%</td>\n",
       "      <td>all credits at this bank paid back duly:4.93%</td>\n",
       "      <td>no credits taken/ all credits paid back duly:3...</td>\n",
       "      <td>existing credits paid back duly till now:53.73%</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>delay in paying off in the past:8.00%</td>\n",
       "      <td>all credits at this bank paid back duly:4.93%</td>\n",
       "      <td>no credits taken/ all credits paid back duly:3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purpose</th>\n",
       "      <td>object</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>10</td>\n",
       "      <td>radio/television:27.47%</td>\n",
       "      <td>car (new):25.33%</td>\n",
       "      <td>furniture/equipment:18.40%</td>\n",
       "      <td>business:9.33%</td>\n",
       "      <td>car (used):9.20%</td>\n",
       "      <td>education:5.07%</td>\n",
       "      <td>repairs:2.67%</td>\n",
       "      <td>domestic appliances:1.07%</td>\n",
       "      <td>others:0.93%</td>\n",
       "      <td>retraining:0.53%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>credit.amount</th>\n",
       "      <td>int64</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>700</td>\n",
       "      <td>3207.35</td>\n",
       "      <td>2731.93</td>\n",
       "      <td>250</td>\n",
       "      <td>417.33</td>\n",
       "      <td>906.3</td>\n",
       "      <td>2301.5</td>\n",
       "      <td>3956.5</td>\n",
       "      <td>7179.4</td>\n",
       "      <td>12715.2</td>\n",
       "      <td>15672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>savings.account.and.bonds</th>\n",
       "      <td>object</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>5</td>\n",
       "      <td>... &lt; 100 DM:60.93%</td>\n",
       "      <td>unknown/ no savings account:17.47%</td>\n",
       "      <td>100 &lt;= ... &lt; 500 DM:10.53%</td>\n",
       "      <td>500 &lt;= ... &lt; 1000 DM:6.00%</td>\n",
       "      <td>... &gt;= 1000 DM:5.07%</td>\n",
       "      <td>... &lt; 100 DM:60.93%</td>\n",
       "      <td>unknown/ no savings account:17.47%</td>\n",
       "      <td>100 &lt;= ... &lt; 500 DM:10.53%</td>\n",
       "      <td>500 &lt;= ... &lt; 1000 DM:6.00%</td>\n",
       "      <td>... &gt;= 1000 DM:5.07%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>present.employment.since</th>\n",
       "      <td>object</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>5</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years:32.53%</td>\n",
       "      <td>... &gt;= 7 years:24.93%</td>\n",
       "      <td>... &lt; 1 year:18.67%</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years:17.60%</td>\n",
       "      <td>unemployed:6.27%</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years:32.53%</td>\n",
       "      <td>... &gt;= 7 years:24.93%</td>\n",
       "      <td>... &lt; 1 year:18.67%</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years:17.60%</td>\n",
       "      <td>unemployed:6.27%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>installment.rate.in.percentage.of.disposable.income</th>\n",
       "      <td>int64</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>4</td>\n",
       "      <td>2.94533</td>\n",
       "      <td>1.13493</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>personal.status.and.sex</th>\n",
       "      <td>object</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>3</td>\n",
       "      <td>male : single:54.40%</td>\n",
       "      <td>female : divorced/separated/married:36.27%</td>\n",
       "      <td>male : married/widowed:9.33%</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>male : single:54.40%</td>\n",
       "      <td>female : divorced/separated/married:36.27%</td>\n",
       "      <td>male : married/widowed:9.33%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>other.debtors.or.guarantors</th>\n",
       "      <td>object</td>\n",
       "      <td>750</td>\n",
       "      <td>0.00%</td>\n",
       "      <td>3</td>\n",
       "      <td>none:90.80%</td>\n",
       "      <td>guarantor:4.93%</td>\n",
       "      <td>co-applicant:4.27%</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>none:90.80%</td>\n",
       "      <td>guarantor:4.93%</td>\n",
       "      <td>co-applicant:4.27%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      type  size missing  \\\n",
       "status.of.existing.checking.account                 object   750   0.00%   \n",
       "duration.in.month                                    int64   750   0.00%   \n",
       "credit.history                                      object   750   0.00%   \n",
       "purpose                                             object   750   0.00%   \n",
       "credit.amount                                        int64   750   0.00%   \n",
       "savings.account.and.bonds                           object   750   0.00%   \n",
       "present.employment.since                            object   750   0.00%   \n",
       "installment.rate.in.percentage.of.disposable.in...   int64   750   0.00%   \n",
       "personal.status.and.sex                             object   750   0.00%   \n",
       "other.debtors.or.guarantors                         object   750   0.00%   \n",
       "\n",
       "                                                    unique  \\\n",
       "status.of.existing.checking.account                      4   \n",
       "duration.in.month                                       32   \n",
       "credit.history                                           5   \n",
       "purpose                                                 10   \n",
       "credit.amount                                          700   \n",
       "savings.account.and.bonds                                5   \n",
       "present.employment.since                                 5   \n",
       "installment.rate.in.percentage.of.disposable.in...       4   \n",
       "personal.status.and.sex                                  3   \n",
       "other.debtors.or.guarantors                              3   \n",
       "\n",
       "                                                                                       mean_or_top1  \\\n",
       "status.of.existing.checking.account                                      no checking account:39.20%   \n",
       "duration.in.month                                                                            20.548   \n",
       "credit.history                                      existing credits paid back duly till now:53.73%   \n",
       "purpose                                                                     radio/television:27.47%   \n",
       "credit.amount                                                                               3207.35   \n",
       "savings.account.and.bonds                                                       ... < 100 DM:60.93%   \n",
       "present.employment.since                                                  1 <= ... < 4 years:32.53%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                          2.94533   \n",
       "personal.status.and.sex                                                        male : single:54.40%   \n",
       "other.debtors.or.guarantors                                                             none:90.80%   \n",
       "\n",
       "                                                                                          std_or_top2  \\\n",
       "status.of.existing.checking.account                                                 ... < 0 DM:27.60%   \n",
       "duration.in.month                                                                              11.941   \n",
       "credit.history                                      critical account/ other credits existing (not ...   \n",
       "purpose                                                                              car (new):25.33%   \n",
       "credit.amount                                                                                 2731.93   \n",
       "savings.account.and.bonds                                          unknown/ no savings account:17.47%   \n",
       "present.employment.since                                                        ... >= 7 years:24.93%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                            1.13493   \n",
       "personal.status.and.sex                                    female : divorced/separated/married:36.27%   \n",
       "other.debtors.or.guarantors                                                           guarantor:4.93%   \n",
       "\n",
       "                                                                              min_or_top3  \\\n",
       "status.of.existing.checking.account                              0 <= ... < 200 DM:27.07%   \n",
       "duration.in.month                                                                       4   \n",
       "credit.history                                      delay in paying off in the past:8.00%   \n",
       "purpose                                                        furniture/equipment:18.40%   \n",
       "credit.amount                                                                         250   \n",
       "savings.account.and.bonds                                      100 <= ... < 500 DM:10.53%   \n",
       "present.employment.since                                              ... < 1 year:18.67%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                      1   \n",
       "personal.status.and.sex                                      male : married/widowed:9.33%   \n",
       "other.debtors.or.guarantors                                            co-applicant:4.27%   \n",
       "\n",
       "                                                                                           1%_or_top4  \\\n",
       "status.of.existing.checking.account                 ... >= 200 DM / salary assignments for at leas...   \n",
       "duration.in.month                                                                                   6   \n",
       "credit.history                                          all credits at this bank paid back duly:4.93%   \n",
       "purpose                                                                                business:9.33%   \n",
       "credit.amount                                                                                  417.33   \n",
       "savings.account.and.bonds                                                  500 <= ... < 1000 DM:6.00%   \n",
       "present.employment.since                                                    4 <= ... < 7 years:17.60%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                                  1   \n",
       "personal.status.and.sex                                                                          None   \n",
       "other.debtors.or.guarantors                                                                      None   \n",
       "\n",
       "                                                                                          10%_or_top5  \\\n",
       "status.of.existing.checking.account                                                              None   \n",
       "duration.in.month                                                                                   8   \n",
       "credit.history                                      no credits taken/ all credits paid back duly:3...   \n",
       "purpose                                                                              car (used):9.20%   \n",
       "credit.amount                                                                                   906.3   \n",
       "savings.account.and.bonds                                                        ... >= 1000 DM:5.07%   \n",
       "present.employment.since                                                             unemployed:6.27%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                                  1   \n",
       "personal.status.and.sex                                                                          None   \n",
       "other.debtors.or.guarantors                                                                      None   \n",
       "\n",
       "                                                                                     50%_or_bottom5  \\\n",
       "status.of.existing.checking.account                                                            None   \n",
       "duration.in.month                                                                                18   \n",
       "credit.history                                      existing credits paid back duly till now:53.73%   \n",
       "purpose                                                                             education:5.07%   \n",
       "credit.amount                                                                                2301.5   \n",
       "savings.account.and.bonds                                                       ... < 100 DM:60.93%   \n",
       "present.employment.since                                                  1 <= ... < 4 years:32.53%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                                3   \n",
       "personal.status.and.sex                                                                        None   \n",
       "other.debtors.or.guarantors                                                                    None   \n",
       "\n",
       "                                                                                       75%_or_bottom4  \\\n",
       "status.of.existing.checking.account                                        no checking account:39.20%   \n",
       "duration.in.month                                                                                  24   \n",
       "credit.history                                      critical account/ other credits existing (not ...   \n",
       "purpose                                                                                 repairs:2.67%   \n",
       "credit.amount                                                                                  3956.5   \n",
       "savings.account.and.bonds                                          unknown/ no savings account:17.47%   \n",
       "present.employment.since                                                        ... >= 7 years:24.93%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                                  4   \n",
       "personal.status.and.sex                                                                          None   \n",
       "other.debtors.or.guarantors                                                                      None   \n",
       "\n",
       "                                                                           90%_or_bottom3  \\\n",
       "status.of.existing.checking.account                                     ... < 0 DM:27.60%   \n",
       "duration.in.month                                                                      36   \n",
       "credit.history                                      delay in paying off in the past:8.00%   \n",
       "purpose                                                         domestic appliances:1.07%   \n",
       "credit.amount                                                                      7179.4   \n",
       "savings.account.and.bonds                                      100 <= ... < 500 DM:10.53%   \n",
       "present.employment.since                                              ... < 1 year:18.67%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                      4   \n",
       "personal.status.and.sex                                              male : single:54.40%   \n",
       "other.debtors.or.guarantors                                                   none:90.80%   \n",
       "\n",
       "                                                                                   99%_or_bottom2  \\\n",
       "status.of.existing.checking.account                                      0 <= ... < 200 DM:27.07%   \n",
       "duration.in.month                                                                              60   \n",
       "credit.history                                      all credits at this bank paid back duly:4.93%   \n",
       "purpose                                                                              others:0.93%   \n",
       "credit.amount                                                                             12715.2   \n",
       "savings.account.and.bonds                                              500 <= ... < 1000 DM:6.00%   \n",
       "present.employment.since                                                4 <= ... < 7 years:17.60%   \n",
       "installment.rate.in.percentage.of.disposable.in...                                              4   \n",
       "personal.status.and.sex                                female : divorced/separated/married:36.27%   \n",
       "other.debtors.or.guarantors                                                       guarantor:4.93%   \n",
       "\n",
       "                                                                                       max_or_bottom1  \n",
       "status.of.existing.checking.account                 ... >= 200 DM / salary assignments for at leas...  \n",
       "duration.in.month                                                                                  72  \n",
       "credit.history                                      no credits taken/ all credits paid back duly:3...  \n",
       "purpose                                                                              retraining:0.53%  \n",
       "credit.amount                                                                                   15672  \n",
       "savings.account.and.bonds                                                        ... >= 1000 DM:5.07%  \n",
       "present.employment.since                                                             unemployed:6.27%  \n",
       "installment.rate.in.percentage.of.disposable.in...                                                  4  \n",
       "personal.status.and.sex                                                  male : married/widowed:9.33%  \n",
       "other.debtors.or.guarantors                                                        co-applicant:4.27%  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toad.detector.detect(data_tr).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "### II. Feature selection, fine classing, and WOE transformation\n",
    "\n",
    "Toad can be used to filter abundant features such as the ones with high missing rate, low iv, and highly correlated features. It can also fine class features with vairous binning techniques and apply WOE transformation.\n",
    "\n",
    "----\n",
    "1. **toad.selection.select()**:  used to filter features based on missing percentage, iv (with 20 bins), and multicolleanrity (with VIF and/or intercorrelation) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(750, 16)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'empty': array([], dtype=float64),\n",
       " 'iv': array(['installment.rate.in.percentage.of.disposable.income',\n",
       "        'present.residence.since',\n",
       "        'number.of.existing.credits.at.this.bank', 'job',\n",
       "        'number.of.people.being.liable.to.provide.maintenance.for',\n",
       "        'telephone'], dtype=object),\n",
       " 'corr': array([], dtype=object)}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The filter criteria include missing rate >=0.5, iv <= 0.05, correlation >= 0.7 (the one with the hightest iv will be kept), \n",
    "# and \"exclude = ['type']\" specifies that 'type' tag will be not dropped\n",
    "selected_data, drop_lst= toad.selection.select(data_tr,target = 'creditability', empty = 0.5, iv = 0.05, corr = 0.7, return_drop=True, exclude=['type'])\n",
    "\n",
    "selected_test = data_ts[selected_data.columns]\n",
    "\n",
    "print(selected_data.shape)\n",
    "drop_lst # As shown, 8 features have been dropped due to low iv. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------\n",
    "2. **toad.quality(dataframe, target)**: return the quality of each feature, incl. iv, gini, and entropy. The output provides information of which features are potentially more useful. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>iv</th>\n",
       "      <th>gini</th>\n",
       "      <th>entropy</th>\n",
       "      <th>unique</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>status.of.existing.checking.account</th>\n",
       "      <td>0.666012</td>\n",
       "      <td>0.368037</td>\n",
       "      <td>0.545196</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>duration.in.month</th>\n",
       "      <td>0.354784</td>\n",
       "      <td>0.406755</td>\n",
       "      <td>0.609659</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>credit.amount</th>\n",
       "      <td>0.351455</td>\n",
       "      <td>0.408680</td>\n",
       "      <td>0.610864</td>\n",
       "      <td>921.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>credit.history</th>\n",
       "      <td>0.293234</td>\n",
       "      <td>0.394090</td>\n",
       "      <td>0.580631</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age.in.years</th>\n",
       "      <td>0.211197</td>\n",
       "      <td>0.414339</td>\n",
       "      <td>0.610863</td>\n",
       "      <td>53.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>savings.account.and.bonds</th>\n",
       "      <td>0.196010</td>\n",
       "      <td>0.404838</td>\n",
       "      <td>0.591377</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           iv      gini   entropy  unique\n",
       "status.of.existing.checking.account  0.666012  0.368037  0.545196     4.0\n",
       "duration.in.month                    0.354784  0.406755  0.609659    33.0\n",
       "credit.amount                        0.351455  0.408680  0.610864   921.0\n",
       "credit.history                       0.293234  0.394090  0.580631     5.0\n",
       "age.in.years                         0.211197  0.414339  0.610863    53.0\n",
       "savings.account.and.bonds            0.196010  0.404838  0.591377     5.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quality = toad.quality(data,'creditability')\n",
    "quality.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__________\n",
    "---\n",
    "First, Combiner() to bin and set bins\n",
    "\n",
    "3. **toad.transform.Combiner()**: combiner is used to fine class numerical and categorical features with binning. We support Chi-squared binning, decision tree binning, binning by step, and binning by quantile. _The following demonstrates the procedure._\n",
    "\n",
    "3.1. **combiner().fit(**data, y = 'target', method = 'chi', min_samples = None, n_bins = None** )**: fit binning. Method supports 'chi', 'dt', 'percentile', and 'step'.\n",
    "\n",
    "3.2. **combiner().set_rules(**dict**)**: used to set bins.\n",
    "\n",
    "3.3. **combiner().transform(**data**)**: transform the features of data into binned groups. \n",
    "____\n",
    "Second, WOETransformer() to WOE transform\n",
    "\n",
    "4. **toad.transform.WOETransformer()**: apply WOE transformation after binnning\n",
    "\n",
    "4.1 **WOETransformer().fit_transform(**data, y_true, exclude = None**)**: WOE transform data by data. \"exclude\" excludes the columns not to be transformed \n",
    "\n",
    "4.2 **WOETransformer().transform(**data**)**: transform data after fit_transform. Normally applied to test/validation data after fit_transform(training).\n",
    "\n",
    "-----\n",
    "Use plots to tune bins. \n",
    "\n",
    "5. **toad.plot.badrate_plot(**data,target = 'target', x = None, by = None**)**: plot the bad rate of each bin across different sets. Different sets can be train/test, or different month etc. \"by\" is name of the column to plot. \"x\" is the column used for comparison (e.g.month, test/train).\n",
    "6. **toad.plot.proportion_plot(**data[col]**)**: plot the proportion each bin of a binned feature.\n",
    "\n",
    "Note: the guide below is the binning as a whole, please refer to the [complete tutorial](complete tutorial.html) for bins adjustment. (TBA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "status.of.existing.checking.account: [['no checking account'], ['... >= 200 DM / salary assignments for at least 1 year'], ['0 <= ... < 200 DM'], ['... < 0 DM']]\n",
      "credit.amount: [2145, 3914]\n",
      "duration.in.month: [9, 12, 18, 33]\n"
     ]
    }
   ],
   "source": [
    "# Step 1: initialise a combiner class\n",
    "combiner = toad.transform.Combiner()\n",
    "\n",
    "# Step 2: fit data and specify binningm method along with other parameters (optional)\n",
    "combiner.fit(selected_data,y='creditability',method='chi',min_samples =  0.05,exclude='type') # fit binning \n",
    "\n",
    "# Step 3: save bins. The bins are saved as a dictionary. \n",
    "bins = combiner.export() \n",
    "\n",
    "# check bins by features \n",
    "print('status.of.existing.checking.account:',bins['status.of.existing.checking.account'])\n",
    "print('credit.amount:',bins['credit.amount'])\n",
    "print('duration.in.month:', bins['duration.in.month'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'status.of.existing.checking.account': [['no checking account'],\n",
       "  ['... >= 200 DM / salary assignments for at least 1 year'],\n",
       "  ['0 <= ... < 200 DM'],\n",
       "  ['... < 0 DM']],\n",
       " 'duration.in.month': [9, 12, 18, 33],\n",
       " 'credit.history': [['critical account/ other credits existing (not at this bank)'],\n",
       "  ['existing credits paid back duly till now'],\n",
       "  ['delay in paying off in the past'],\n",
       "  ['all credits at this bank paid back duly',\n",
       "   'no credits taken/ all credits paid back duly']],\n",
       " 'purpose': [['domestic appliances', 'car (used)', 'retraining'],\n",
       "  ['radio/television'],\n",
       "  ['furniture/equipment'],\n",
       "  ['repairs', 'business', 'car (new)'],\n",
       "  ['education', 'others']],\n",
       " 'credit.amount': [2145, 3914],\n",
       " 'savings.account.and.bonds': [['unknown/ no savings account'],\n",
       "  ['... >= 1000 DM'],\n",
       "  ['500 <= ... < 1000 DM'],\n",
       "  ['... < 100 DM'],\n",
       "  ['100 <= ... < 500 DM']],\n",
       " 'present.employment.since': [['4 <= ... < 7 years'],\n",
       "  ['... >= 7 years'],\n",
       "  ['1 <= ... < 4 years'],\n",
       "  ['... < 1 year'],\n",
       "  ['unemployed']],\n",
       " 'personal.status.and.sex': [['male : married/widowed'],\n",
       "  ['male : single'],\n",
       "  ['female : divorced/separated/married']],\n",
       " 'other.debtors.or.guarantors': [['guarantor', 'none', 'co-applicant']],\n",
       " 'property': [['real estate'],\n",
       "  ['building society savings agreement/ life insurance'],\n",
       "  ['car or other, not in attribute Savings account/bonds'],\n",
       "  ['unknown / no property']],\n",
       " 'age.in.years': [26, 28, 35, 39, 49],\n",
       " 'other.installment.plans': [['none'], ['bank', 'stores']],\n",
       " 'housing': [['own'], ['for free'], ['rent']],\n",
       " 'foreign.worker': [['no', 'yes']]}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1a2031e390>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAAF2CAYAAACPsxINAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5xsZ13n+09VdVX1dV/TJJsdcjGYJw4YxYHBEZkEFeR4wTnIAYbMRESIIDAQ0DPIbZRoBjlIQEZwgkBAGT06ozhHuQ0qF0eNEKNBcB4EsxMhQDabXHbvvtT1/FFV3XVZ1V2ru6qruvvzfr36tVdXrVrr6UB21ref3/N7MvV6HUmSJEnSYLLjHoAkSZIk7SWGKEmSJElKwRAlSZIkSSkYoiRJkiQpBUOUJEmSJKUwNe4BbEMReAzwFaA65rFIkiRpf8oBJ4BPAWtjHosmzF4MUY8BPjnuQUiSJOlAeDzwZ+MehCbLXgxRXwG4775z1Gq7u8fV8ePznDmztKv3lCRJOujG8QyWzWY4enQOms+eUruRhqgQwnOBlwBngWfFGE+1vfeTwKuBrzdfuibG+LkBLlsFqNXqux6iWveVJEnS7hrjM5jLR9RjZCEqhLAIvAK4ErgKeBPw1LZTDgOvjjG+Z1RjkCRJkqRhG2V3vicBt8UYl4EPA48LIWTa3j8M3D/C+0uSJEnS0I0yRJ0AIkCMsQbcBxxre/8w8PIQwt+FEN4SQsiNcCySJEmSNBSjbizRHtIWgPZi1v+n+ecy8AfAvwF+c9ALHz8+v+PBbcfi4sJY7itJknSQ+QymSTLKEHUP8FiAEMICcJTGbBQAMca7WschhN8Frkhz8TNnlnZ9geHi4gKnT5/d1XtKkiQddON4BstmM2P7pb0m3yhD1EeA14UQZoGrgQ8CTw8hPBT4VeBHgd8GZoHvA94xwrFIkiRJ0lCMbE1UjPHrwI3ArcDPAi8HTgKXAhXgsuZ7nwP+Lsb4/lGNRZIkSZKGJVOv77l9jy4B7rScT5Ik6WAYcznfpcCpXb25Jt4ou/NJkiRJO5LLZbj/vgfI5TJbnyztEkOUJEmSJlIul2F1+Rw/+5JfYHX5nEFKE8MQJUmSpInTClDXXXM9/+vjf8V111xvkNLEGPU+UZIkSdrH6vU6lUqF0lqZtbU1Smsl1tZKjePVtuO10vp7peZr68erjeNyqfH+4aOHuPYnnsbPvPDnuPvUlwG4+9SXue6a67n5fTcxPTtHtbrn1vVrHzFESZIk7SPVSrUzoKyt9Qac1a6w0y/grLYdl1qBpzcE1Wq1bY+3UCxQLBaafxYpFgv8+POfya+84R3rAarl7lNf5oZXvpEb3vizkMnv9B+VtG1250vB7nySJCmNarVKqVSmtLbG2morpKw1w0uJ0uoapVK5J7R0Bpxy52c6zm2e03aParW67fHmC3mKzTBTWA82nQGnUCxQnG4cN85vvj7ddU6xSHG6QKGwcZ3idLHjWvlCnkymtzyvvZSvPUhddMnJXZuJsjufNuNMlCRJOhBqtRql0sZMSqm0EWS6A0rjnOSA0z3D01Oa1lbGVqlUtj3eqampjnCyHkSKRaanpzl0eKHn9ca5zeNCnkIzyDSCTbH5XoHCdOP99c83w042OxnL5avVOtOzc9z8vpvWg9RuBihpK4YoSZK06+r1OuVSuaOcrLUuZj3gJM3KtM2+rH8+oTRt/TptAadcKm97vLlcrhE0ihvhYyPgFJhbmNsIKMUihWK+Z1amJ+BMFzs+U2yFmeb7uVxuiP/E9572IHXDK9/Ia278aQOUJoblfClYzidJ2o/q9TrlcmWjTKx7bUzXGpjyemDpLU3rWDuzRcDZrmw22zn7sh4+GrMpibMybWVp/UrT1sNOMU/3DE9u6mAHmnHK5TJkM1Vq9dyuBijL+bQZZ6IkSZowlUolcRamvNZWWtY1c9P+emmLgLPx+Y1ztvtL1Uwm01jbsh44OsvGDh05tH6cbwWc6WJvCEoKOAlrbArFPFNTU4nraLQ/Vat1ji0e9hfZmiiGKEmSNtHqdNZaG9Pdsnl9tqVU7llPs1nA2Shha1tH07xHrTrcTmetcDK/MEeheKxjXcz6cWFjViZxlqfQFpDa1tjk8wYaSQePIUqStGfUarWNGZa2TmelUjl5VmZ1I5wkBZyOTmdJHdCG0OlsvbxsemOGplAsMDs7y9GjR/p3QGufhSkUEkvTWoFnI+AUDDSStAsMUZKkbanVar2NARI6nZW6S9ASA85gpWnl8nA7nbXCx/R0MbnTWTHf2dGsaxamUMhvOnMzKZ3OJEnDZYiSpH0gqdNZUtlYcueyhNK01jnrm2v2BpyddDrL5rK9+8m0dzqbn+3cZ6a7EUBSwOnqdNbdHe2gdzqTJA2PIUqShqxerzcaA/TZV2ZttbN7WdIeNd3d0foFnNY9SqXtNwbo7nSWL3Zutnn4yKGOTmfta2f6bcLZvf+Mnc4kSfuJIUrSvtfe6azc2jhztbdsbG2L9sx92z93dTorlcrUattvDNDZmrmr09nhhbaZmN5OZ+trY5ICTltzgPZ72OlMkqR0DFGSdlW1Wu27cWapVE6clenodFYqJZambZSwlXteH0qns0KhJ7TMzc9y9PjRtrDSZ1amPeD0lKa1rbEpFMgX8gYaSZImnCFKOsDaO531LP7vWgOT2Lmsa/alc+1Nbwe00lqZSmX7jQHy+amNWZn2TmeFAjMzMxw5crgztGyycWZ3p7OkEJQv5G0MIEmSehiipAlRr9cTF/mX1rrKxhLW2HTPvnSHoI3NNTvbQu+001li57JmwFk4PN8IKIV8T0ezpMX/SaVp7c0DCoW8jQEkSdJEMERJCVqdzlrho3NfmbYNMkudoaVfCOpZT5MYcErbHm9Sp7NG+Ggc9+t0tmnAaWsOkLRnjY0BJEnSQWWI0sTr2+msVOpcT9PV6aw74HR3OtsIPF1lZzvsdJbJZDrWunTvH3P4yKG20rHOtTOt443NOZP3nykWi+RbAWe60RhAkiRJu8MnrwHlchnuv+8Bcrkc1er2Hq73i2qlmtyyua05QL9Zmdbr7a2ZkzbhLHetzxlWp7P1zmXNgLJwaJ7jXZ3OugNOd6ezVnOAfmts7HQmSZK0vxmiBpDLZVhdPscNr3wjr7nxp5menZuYINXT6ax78f9q78L+pI5m7Z3OOgNObwe0nXQ6a8yeJLdnnp2b5eixI5vuP9Oz+D9hE87uxgAGGkmSJA2TIWoLrQB13TXXc/epL3PdNddz8/tuSgxStVqtEUD67D9TKrUFnNWuErKk7mgJnc7WO6A1XxtWp7P2gFIsFpmenubw4UPrAWd948yujmbdi/+TZm7sdCZJkqT9xBC1ie4ABawHqV9662t54y+8jTu/cPf6+ptyqbyDe+U6WjZ3di4rMH9ovrMDWlsjgE0DznRb84C2ttB2OpMkSZK2xxC1iVqlxA2vfON6gGq5+9SX+c9vfCcvfcVP8lvveX/XepqEgJNUmtZaU9MMQHY6kyRJkvaGzHY7kI3RJcCdZ84sUauNduxJM1EAF11ysm9JnyRJkoZrcXGB06fP7uo9s9kMx4/PA1wKnNrVm2viuUBlE9VqnenZOW5+301cdMlJwAAlSZIkHXSGqC20B6nHXfUvDFCSJEnSAWc534ByuQzZTJVa3X2iJEmSdpPlfJo0zkQNqFqtc+ToYQOUJEmSdMAZoiRJkiQpBUOUJEmSJKVgiJIkSZKkFAxRkiRJkpSCIUqSJEmSUjBESZIkSVIKhihJkiRJSsEQJUmSJEkpGKIkSZIkKQVDlCRJkiSlYIiSJEmSpBQMUZIkSZKUgiFKkiRJklIwREmSJElSCoYoSZIkSUrBECVJkiRJKRiiJEmSJCkFQ5QkSZIkpWCIkiRJkqQUDFGSJEmSlIIhSpIkSZJSMERJkiRJUgqGKEmSJElKwRAlSZIkSSkYoiRJkiQpBUOUJEmSJKVgiJIkSZKkFAxRkiRJkpSCIUqSJEmSUjBESZIkSVIKhihJkiRJSsEQJUmSJEkpTI3y4iGE5wIvAc4Cz4oxnko4503Ao2KMTxjlWCRJkiRpGEY2ExVCWAReATwWuAF4U8I5VwDfPaoxSJIkSdKwjbKc70nAbTHGZeDDwONCCJmuc34ZeP0IxyBJkiRJQzXKEHUCiAAxxhpwH3Cs9WYI4YeAfwJuG+EYJEmSJGmoRromis6QtgDUAUIIBeBVwA8B89u58PHj2/rYji0uLozlvpIkSQeZz2CaJKMMUffQWA9FCGEBOEpjNgoa66COA38IFIHLQgg3xRivH/TiZ84sUavVhzviLSwuLnD69NldvackSdJBN45nsGw2M7Zf2mvyjTJEfQR4XQhhFrga+CDw9BDCQ2OMNwGXA4QQLgZuSROgJEmSJGlcRrYmKsb4deBG4FbgZ4GXAyeBS0d1T0mSJEkatUy9vrslcUNwCXCn5XySJEkHw5jL+S4FTu3qzTXxRtmdT5IkSZL2HUOUJEmSJKVgiJIkSZKkFAxRkiRJkpSCIUqSJEmSUjBESZIkSVIKhihJkiRJSsEQJUmSJEkpGKIkSZIkKQVDlCRJkiSlYIiSJEmSpBQMUZIkSZKUgiFKkiRJklIwREmSJElSCoYoSZIkSUrBECVJkiRJKRiiJEmSJCkFQ5QkSZIkpWCIkiRJkqQUDFGSJEmSlIIhSpIkSZJSMERJkiRJUgqGKEmSJElKwRAlSZIkSSkYoiRJkiQpBUOUJEmSJKVgiJIkSZKkFAxRkiRJkpSCIUqSJEmSUjBESZIkSVIKhihJkiRJSsEQJUmSJEkpGKIkSZIkKQVDlCRJkiSlYIiSJEmSpBQMUZIkSZKUgiFKkiRJklIwREmSJElSCoYoSZIkaQ8LIfx+COGqIVznurbja0MI/24I1/y/Qwjft9Pr7HAMh0IIz2j7/t0hhB/ZyTWndj4sSZIkSXtZCGEWeAFwM0CM8b3DuG6M8Q3DuM4OXQg8E/h/h3VBQ5QkSZK0x4QQXgi8BLgLOL/52n0xxqPN43cD748x/kEI4Xbg14GfBL4LeBnwfwDHgVfHGH8H+JPGx8LtwAuBxwBHYow/H0L4Jhrh6iHA54CfjDE+0LzH54AfAC4H/q8Y4593jfMm4G9ijO8JIdwBvBtozXA9McZ4puv8O4D3As8G/jfwa8DPNcf6xBjjl9KMB7gD+ANgsfmz/Z/NWz0+hPBi4FuSxr0Vy/kkSZKkPSSE8AgaQefRwA8C57b4yBHgaIzxyhjjEvBHMcZ/CTwJeH3znGcAMcb4qIRA8R7gLTHGK4G/Bm5oe+9S4HuAnwdetMU4FoB7YozfQSMg/Wifc+4Evo1GEPrhGON30wh5z0o7nubP+1zgT5s/26nmebPAEwccdw9DlCRJkrS3PAH43RjjgzHGEvDVAT7ztrbjO0MILwVeB5zc7EMhhDngEcAHmi+9A/j+tlM+HGOs0wgzFwwwjg81/9zs/I/GGKs0ZpH+pPna7cAFQxxP2nF3MERJkiRJe0uexoxNt82e7asAIYQFGsHkFI01UMtb3Csz4H3Kfc7tZ5DzywnnD3s8acfd94aSJEmSJtetwJNDCMUQwmHgm5uv3xVC+JYQwgng8X0+ezlQiTG+H7gEmGm+fi9wPITQkQ+a5XB/T2MNFTRK4z6y2eBCCM8IIVw/6A8TQnhDCOFfDXLudsYDfJktZtzSMkRJkiRJe0hzzdL/oFHu9hs01hBBo3nEx4C3An/U5+N3AP8YQvgb4Cdan40xrgAfBT4XQnhy12euBV7WbPrwL4BXbTHEkzTWJg3qchpNIgaVajwxxi8AD4QQPhNC+LYU9+krU6/Xh3Gd3XQJcOeZM0vUars79sXFBU6fPrur95QkSTrIpqhRq1TJTuWo7OLv/7PZDMePz0MjDJzatRtrT7DFuSRJkratXq9Tr9Ubf47g+LyHHOLGH3s9r3nfq8gUiuP+cSXAECVJkoYo1QNyrTayB+9a69rN7xniPWq1WvJ7m3yWgce92Xl97tvnuPtnrtVGE3YYUVXTc1/348wemuXeu74GwP2n7+fI4hHyhfyuzkhJSQxRkqSRWH+IG/IDcm0ED5eDPcCmvF9PYBj2w2ut914jmQ1I+c/2IMtkyLS+sts/zmY3P48+r2cz2T7X3Xg9u81xMcA52Wx2Rz939/F5Fy7y+ue8Yf0f71tf+qsAzkhpIrgmakDjqseV9otBH6iTH5B38HCZ9IA5qofZtA/Vgx53/UZ9aMejum4zjLD3/vsyXH0eqLd6QE573P5Anc1mh3bdnuNsvwf07vO2cf1JGPcOw07rfwcNzxQ1yqUy95++n7e+9Fd58ZtfuKszUa6J0maciRpQuVTmhmt+0d9+DMnQH9xGWBIyugfYtnKT5veDhIxBj7tLWfods80H5Ekp99grdv7gtv3f8GazWTJT43qA3eqc7T/gZjf5jfxO75fNpr1u57EP1NLOVciSKRQ5sngEgCOLR8gUilTGPC4JDFFbWv8tyL33A3Dmnq9z6NghavU68fYvbvIAW2db5R4DPFCn+019ujKM3SoJOeiG+YC89W94ux48Byj3GPQBNvHBcUe/pc5uPMDu8Dfy/cc9nt94S5K2J1/I83O/9ZpmNZA0GSzn20K9tMYN1/xiz+v//s0v5Featbk7ktn+g1uaB820v60dxoNmvwfqYddMdwaGvVESIkmSBjeObWYmvZwvhPBc4CXAWeBZMcZT4x3RwWKI2kJ3Pe6LfvkFHF48QjaX5f77lndUEtIKNpIkSepvL4eob9zx6WcBNwIXAXcDrzx25aP/607GFkJYBP4CuBK4CnhejPGpO7mm0rGcbwvd9bhHLzi2vibq6PmujZIkSVKyZoB6BzDbfOli4B3fuOPT7DBIPQm4Lca4HEL4MHBLCCETY9xzsyN7lSFqQNbjSpIkqd037vj0tcBzNjnlO4Hu37rPAu/6xh2fvq7PZ9517MpHv3eLW58AIkCMsRZCuA84BpzZetQaBnt1D6hCluMnjtneXJIkSYPqV7Y0jHKm9ofSBcBZqF3kTJQkSZK0Dc0Zo76zRt+449OnaJTwdbvr2JWPvnoHt74HeCxACGEBOArct4PrKSWnVSRJkqTReCWw3PXacvP1nfgI8KgQwixwNfBB10PtLkOUJEmSNALN5hHPA+6iUW53F/C8nXbnizF+nUbHv1uBnwVevsOhKiVbnKcwjvaakiRJB91ebnGu/SnVmqgQwpEY4/0pzu+7CVgI4UXAjwHnAb8RY3xtmrFIkiRJ0jgMVM4XGj4HfDaEcDKE8PchhCu2+Mwi8Aoai95uAN7UdcodzfeuAJ4TQnhY6tFLkiRJ0i4bdE3UW2nMKN0bY/xy8/ubt/jM+iZgwIeBx4UQMq03Y4yfADLAtwP30+gyIkmSJEkTbdAQdTzG+D9b38QY3wYc2uIzHZuA0Wi7eKzrnA8A/wv4qRhjdcCxSJIkSdLYDLomqh5CmKa5iVcI4QIgN8DnNt0ELMb4/SGEy4HfCyF8T4zx3gHH01rot+sWFxfGcl9JkqSDzGcwTZJBQ9TbaZTkPSSE8J+AfwP80hafGWgTsBjj50MI/5NG+d9vDjge7M4nSZJ0MIy5O5/UY6AQFWN8ZwjhH4AfBPLA89rL+/r4CPC69k3AgKeHEB4KvAd4LvDLwCyNAPV72/oJJEmSpAMmhPAfgJcCr48xvmXc4zloBgpRIYQbYoyvAT7R9tpbYowv6feZGOPXQwitTcDOAs8Cnkpjn6f7gTngUzTWSb07xvjJ7f4QkiRJ0iT67M2//QTgV4AnAt/SOn7Edc/86g4v/T+Ab9rhNbRNm4aoEMLP0yjDe0YI4XDbW3ng+2l07Osrxvgu4F1tL7W3Of+PzS9JkiRp32kGqD8ECsD7gO9sHr8GeOFOrh1j/PsQgt2tx2SrmahbgccANeBM2+sV4JpRDUqSJEmadJ+9+bevBZ6zySmPBmZobOtzNRtN15732Zt/+xF9PvOuR1z3zPcObZAaiU1DVIzxA8AHQggfjDH+1S6NSZIkSdoP7qBRwneIjQBVA/5hbCPSUAzane++EMJbgHkaSToHPDzG+LiRjUySJEmaYM0Zo76zRm3lfO3b/lSAjz/iumf+1IiHpxEadLPd/0qjfvO7gFPAPwM+M6IxSZIkSfvBr9B4hgZYAUrN7582thFpKAYNUQsxxhfQ2CvqgzS6i/zLkY1KkiRJ2vueCNwMnAauBd7ZPH7GTi4aQjgRQrgdeD7wMyGEj+50oEpn0HK+VlOJLwCPjDF+KoSwuzvdSpIkSXtIs435C9noxPffgB2X8cUYvwI8aqfX0fYNGqK+EEJ4M41Nct8ZQpin0eZckiRJkg6UQcv5XgB8MsZ4O/AO4HuA60Y2KkmSJEmaUIPORP1/McbvBYgxvh14++iGJEmSJEmTa9CZqCMhhLmRjkSSJEmS9oBBZ6LOAXeFEO4AllovxhifMpJRSZIkSdKEGjREvXOko5AkSZKkPWKgEBVjfM+oByJJkiRpa81O2W8DvpXGJr7/Nsb4j+Md1cGyaYgKIdSAfvtB1WOMg85kSZIkSQfK7zz/zV8Fzk9462tP/7WXXrCDSz8SeF+M8cMhhBcCNwDX7OB6SmmrELQIZGj8D3MX8F+AKvBs4OKRjkySJEkH1nQO6pUqAOe+/gDFeuP3+pmpHKvVcY4slaQAtdnrA4kx/mXbt38GPHMn11N6m4aoGOMZgBDCo2OML2h761dCCJ8e6cgkSZJ0YNUrVT702lt6Xn/y654NmdyujyfJ7zz/zdcCz9nmZz/W5613Pf3XXvreFJd6LPDX2xmDtm/QFudzIYTQ+iaE8K1AcTRDkiRJ0kFULZVZ+caDPHD316iWK+MezsQLISwALwPeMu6xHDSDrml6NfCXzRbnWeCfAc8a2agkSZK0L1RLZUpLK/2/zq1QOrtC+dwK1dJGcLr6ZU8b46gH05wx6jtr9DvPf3O/3gI8/ddeevVO7h1CKAD/HbjRphK7b9DufL8XQvgz4LubL30ixvj10Q1LkiRJk6paqjTDz3IzCK02js+tUF5aZW1pmfLSCqWlVaqlcuI18rPTFOanKczPcvhhDyHfPC7MzzS+FmZ2+afaO0IIGeAW4NMxxjSlfxqSgUJUM+k+Dpij0Wjih0MID48xvmqUg5MkSdLuWA9GXTNF5YTZo/7BqEhhfob8/AwLJxcpLMxQmGuFolkKc9ON9+dmyOY2X1WSre+d7hGb+Bp9uvPt8LovoNFM4tYQwu3N154WY/ziDq+rAWXq9b6zjOtCCL8PfBNwAridxgK2j8UY//Voh5foEuDOM2eWqNW2HvswLS4ucPr02V29pyRJ0nZVy5XNA1GzlK50boXqWv9glG8FoU2+BglGabR358tlM1Rru9udL5vNcPz4PMClwKnR31F7yaBror4d+Gbg7cCbaKyLevuoBiVJkqRk1XKF8rkV1s42Q1FbEOqZMeoTjKZmiuvhZ+Hkef2D0ew02anxdMJbrbLehW/xvLZfZO+LCSrtdYOGqK/EGCshhM8Dj4wx/m4I4fAoByZJknRQtIJRaWm1GYCaa42WVnuO+wejwvqaooUTx5vlcxvriwpzjTK7wtz4gpG0XwwaopZCCM8C/hZ4XgjhfwPzoxuWJEnS3larVAfuSldZLSVeY2qmsB6EFk4cozB/kvz8DMX52WYjhplGcDIYSbtq0BD1QuC5wC80//w48MpRDUqSJGkS1SrV3vK55nF5aYW1tnVHfYPRdGGjlO7EscbsUNLX3IzBSJpQg4YogKuAlwN14JPAH41kRJIkSbtoPRgN0JVuq2CUn59h7vxjHH34TE8pXaP5wjS5fJrHL0mTaNB/i/8L8E7g8c3vnw/8OvD9oxiUJEnSTjSC0cZ6onL3WqO2fY0qK/2DUaMr3XQjGF02s76vUWdXOoORdNAM+m/80RjjO9q+f2sI4SdGMSBJkqQkrWDU0ZFu/Xi5GZoaQalfMMoV8+vhZ+78oxydf2hH+VxhYZbC/DT5uRmDkSZWCOEi4D/T2IJoBbg2xvj34x3VwTLo3w5fCCE8NsZ4K0AI4UrAzbwkSdKO1Kq1Zle6rb8qK2uJ1+gIRotHOPpNJzZK6Zoldq1jg5HG4cVXX/9VGpvufu2tH7vpgiFcsgL8dIzx8yGEFwE/DTjBsYs2/ZskhPAZGmugFoA/CyHcQaM7/7cDnxv98CRJ0l4zaDAqn1uhvLxJMJprC0aXnmh2pZvpacRgMNIecH7XnzsSY7wHIISwCFwO3DaM62pwW/2t86JdGYUkSZpoHcGotZ6oa6PXViOGvsGokF9fUzS3eIT8pRdsrC+am26W0jXK6nIFg5Em34uvvv5a4DmbnPJdQJ7GpEQGqL/46uvrQBn48z6feddbP3bTe7e6dwjhXwF/3Px6aZpxa+c2/Rsqxvjx3RqIJEnaXbVqjfLyasIs0XJXI4ZVyiurjcfALrnC1HrJ3Mzxwxy+5IKOUrqOGaNCfvd/SGm8Wv+nz3T9ueN/GWKMnwghzNLYdugNNEr6tEsy9XrC34iT7RLgzjNnlqjVdnfsi4sLnD59dlfvKUlSGt3BqHvvovaNXsvLycEom59K3rfIYKQxGcczWDab4fjxeYBLgVPbuUbbWqg1oNj257DWRhFCOAzcFmN8+DCup8E4Vy5J0oSr12q9Xem6NnpdD00DBKOZ44c4fPH5XV3pWu26Z5gqGoykYWgFpWYJH0DxrR+7KbPJRwYSQngu8MkYYwR+lG2GPG2fIUqSpDGo12qUl9c69y5aWm07bg9Ga5BQOZLN55priqaZObrA4YvOb5bSNdYdtTdgMBhJY/U1mt35hnS9W4G3hRAeCjwAPHtI19WALOdLwXI+SdJmOoNR0ldzfdG5RnOGvsGoa01Rb1e6RnDKFfJkMjv+pbY08fZqOZ/2L2eiBpDLZZjJZzuOl1arYx6VJGk31Gv1jTVGfUrp1tcb9QtGU7mNUrqj8xx62GJiKV1hboZc0WAkSZPOELWFXC7D7BScu+tLFK+4bOP4wpOslWvjHp4kaRvqtTrllaSudL0NGMrLq9QTKh9awSg/P0Px8DwLFy7270pnMJKkfcUQtYWZfJZzd/LaUvcAABqHSURBVH2Jez/1GaZmZ7j3r/6WTC7HicVj3H/rZ8jmp8gU8mTy+cZxPk82nyfTcdz83v+AStLINILR2kbJ3NIKa0vLzWYM7fsaNcrp+gWj/Nw0hVYwOnleW/ncxlojg5EkHWyGqC0srVaZvfhhnDw0zz996BMAXPKU7+X0Zz5PbWWFyoNl6uUy9XJly2tlpqY6wtb6sSFMkhK1glFiV7ruGaM+wSiTy67PCBUPz7UFo7ZSurnGjNLUdMG/ayVJWzJEbSGXy5CrV7nnz/96/bUHT32JY4/+to5yvnq9Tr1coV4uU2uGqvbjxp+dx4YwSQdRvV6nsrJZ84WUwWhhloWHHu8opevoSmcwkiQNmSFqC61yvkwux2VPezL3ff5Olu+5l6OPDKyVN87LZDJkCnko5Mlt4z6GMEl7WSMYlRLbc7fPILXWG/UNRnNtwejE8YSudAYjSdL4GaK2sLRapXjhSS685EKm52eZu+Jyjj4ysFwablOJXQthpTK11nvLy1Sax/WKIUxSp41g1ApCy81Suo29jMqt43Or1Ku9fy+2B6PC/AwLFxxvBqLpZind7Pr7UzMGI0n705UXX/UE4N3Aj99x18f/dFjXbe4T9XngB2OMHx/WdbU1Q9QA1so11sowPb9xPGmGH8K6Zr1aYcsQJu1p9XqdymqppxvdWkJXutK5leRglM10lMzNXXC0Y21ReyMGg5Gkg64ZoP4QmAX+8MqLr/qhIQap1wNfGNK1lIIhSsB+CWGNsNUevAxhOgi6g1G5Tyld67V+wSjfFoTmzj+a2Kq7UUpXJJP13ydJ2kpXgIIhBqkQwncC88DtOxultsMQpaHYEyGs70zXlCFME6cVjAbpSrdlMGrNGD0kIRg1g5PBSJLSu/Liq64FntPn7SPAtwLZrtdngY9eefFVnwHuT/jcu+646+Pv3ey+IYQM8EvAs4HXphmzhsMQpYkwlhB2zhCm3VWv16mulZvBZ7m5tmiFjWYMbWuNzq1Sq1R7rrEejOamKSzMNoJR87jVprvViCE/YzCSpDG6gt4A1ZJtvv+X27z2jwF/GmO8M4SwzUtoJzL1em+HpAl3CXDnmTNL1BK6O43S4uICp0+f3dV7andsJ4R1vG4IO7A6g9HWXen6BqPZ6Z5Zoo2OdLPNTV5nyM9MG4wkHTjjeAbLZjMcPz4PcClwKu3nE0r52i0D2y7pCyH8IXABUAYuA04DT48xfnY711N6zkRJOBOmTuvBqFU+11VK192AISkYkck0ZoiaYWjuvMM9bbpbpXQGI0naf+646+N/euXFV/0QvUFqRwEKIMb4Q63jEMK7gVsMULvLECUNwchDWKm3bX313DKVVjt7Q9iW6vU61VK5qz13/650tXL/YNRqwHD4+OFGGOroStecRZotksn2q+KQJB0ECUFqxwFKk8FyvhQs59Okqtdq1CuVLYNXvw2c93IIq7SV0pXb1xol7GuUHIxoltJtlMy12nM3vp81GEnSmO3Fcr52o9onSuPjTJS0D2SyWTKFAhQK25sJ20YIG81MWKNtfY0M5XKVylqZ8mqZ8kqpsanrekndRlOGWjnh3uvBqBF+Dl90qE9XulmDkSRp5JrB6ZJxj0PDY4iStCshrLKyRnlpheVzK5S/sUx5eY3yaonKWoXyWoVKpUalXKNSqVHvM8ucy2fJF6aYKuaZmclz6NhxCnNFCrPTjfVGC7MUD81SODRHtljc2FNsau+XI0qSpMlhiJK0bY01RoN1pauW+swYzTRmjIrHppmfKZKfKZCfzjNVmGoEpnyW3FSWqWwGqpXkcsTSg/CNB6l9A1ZofPXcaovZr+QNnA1hkiSplyFKUodqqbKxnmhphdK51fW1ReWlVdaWlpvBaJVqqZx4jUYpXWN26NCFDyE/P71ePtcopZtultJNk83tvJRuvOWIhjBJkg4aQ5R0AKwHo64Zo56udEsrmwSj4npXuoWTi71rjFrNF+ZmhhKM0tjba8IMYZIk7TWGKGmPqpYrmweitpK66lpyMJqaKa6Hn4WT501UMNpNuxPCOjdq3ghhZepJ+0x1j9EQJknSxDBESROkWq5QPrfC2tlmKGoLQj0zRmmCUXMGqWOD19lpslPbiQzqNpQQVq4kzHj12bTZECZJ0lgZoqQRawWjVkvu1kav7Zu+to77B6PC+pqihRPHO/Yyys9PU5yfbXSnmzMY7UWZbJZMsUC2WNjW5ycihK0Hr4Q9wgxhkqR9xhAlbUOtUk3uStdVSlc+t0JltZR4janpwsaM0YljFOZPNoJQwn5GBiNtZm+EsM5w1TH7Vdh802ZDmCRp0hiipKZapdpbPtfWpnutbd3RVsEo3wxG+T6ldAYjTZJxhLDKueX1Y0OYJGmvMURpX1sPRgN0pdssGLW60s2df4yjl/VrvjBNLu+/Ujp4DGGSpINmpE98IYTnAi8BzgLPijGeanvvKcB/AI4DvxFj/MVRjkX7RyMYrTZnh5Ypd681atvXqLKSHIxyxXxzTdE0c+cf5ehlJynMT3esNSrMT5OfmzEYSSNmCJMk7TUjezoMISwCrwCuBK4C3gQ8te2UK4DvA+pADCH8bozx86MajyZbrVprNF9IKKXrmTFaWUu8RiMYNWaF5s4/ytH5h3aUz+XnZyguzBiMpH1m90JYmXqp8X5l6dz6ufWqIUySDppRPkk+CbgtxrgcQvgwcEsIIRNjrAPEGN/QOjGE8NfAwwBD1D6yHow2a8AwSDBqldItHuHopSc2yue6yukMRpK2Y7ghrNx1vDEDZgiTpP1jlE+dJ4AIEGOshRDuA44BZ9pPCiFM0Zit+swIx6Ih2TIYtXWlKy/3CUaFthmjZjDq15UuVzAYSZps+y+Edb6fmcoZwiSpy6ifULNtxws0Sve6PR/4kxjjvWkufPz4/E7GtW2Liwtjue8o1ao11pZWWH3gHKtnl1l7cJmVB8+x9uAyq62v5utrSyuJ15gq5pk+NEtxYZajF57H9MIs04c2voqHZpk+NMf0oVmmCvld/gklaf+q12pUS2WqayVqzT+rpTK1UonqWplqqUSt+We1VKa2VqK6ukrlwQeprZWpVSqb3yCTIVdohK1csdA8LpAr5skVCmSbf+aa73efl807E6bh2I/PYNq7Rhmi7gEeCxBCWACOAve1nxBCeBLwY8AT0l78zJklarWkTDY6i4sLnD59dlfvuV21ao3y8upAXenKK6uJ8TZXmNoomzs8z/yFi43SuoRZo9wWwagOrNRh5YFVYHUkP7MkKQ/FPBQ3Xsk2v/r9Bz/NTFitVKG6WqZ2djn9TFhhY4ard9NmZ8K0uXE8g2WzmbH90l6Tb5Qh6iPA60IIs8DVwAeBp4cQHhpjvCmE8GjgbcD3xBiXRjiOfaNeq7V1pWsGolYzhvbjcyuUl5ODUTY/tR58Zo4f4vDF5yeGokGCkSRp7xtLOeLqkiFM0p42shAVY/x6COFG4FaaLc5pdOe7pHnKB4Bl4PdCCDngozHGnxnVeCZVvVajvLy23pJ7s+YLWwej6Y1g1Kcr3VTRYCRJGh5DmKSDKFOv725J3BBcAty5W+V80znW9xDJZTNUm/fMTOVY7fP39nowWg9Ay5S69zJaWqW0tNxovpDwv0E2n1vfx6gRiGabpXTTFOZn2xoxTJMr5P0LXpJ0IA0cwkrJe4dhCNsTxlzOdylwaldvroln67Mt1CtVPvTaW3pef+Krr+HuT/9DTyld+Vxjs9fEYDSV2yilOzrP4YctNmaJOtp1zxqMJEka0HBmwjo3Y66VK9RLyfuF1cuV5kzYgCEsk2m0mTeESfuKIWqbystrfPFDf9UTjA49bLGjRXdhoa1dd9FgJEnSJGmEsCLZYnHrkxMYwqSDyRC1TcWFWR7/qn9nMJIk6QDbEyGsawPmpON+e4dlcoYwKYkhapsyuSxTU7lxD0OSJO1hhrDN5XIZZvLZjuOlfovSpV1kiJIkSdqj9nMIm5ouMjuV49xdX6J4xWXMTtE4vvAka+Xatn5eaVgMUVvITOV48uueDfR258NfhEiSpD1skkPYhd/3XSx9ZZnTf/1ZCgtzfPUvbieTy3HhJReyVt7uTywNhyFqC6tVINMo21s8r629pgFKkiQdcKMMYfd/9T4On1zk5BMey90f/iQAlz71+1ku7bntebQPGaIkSZI0FpuFsFwuw/QU/NMHPrb+2gNfOMXcFZcPsr2XNFLZcQ9AkiRJ6jaTz3Lu7nvI5HJc9rQnc+zKwPI991KYslugxs+ZKEmSJE2cpdUqxQtPcuElFzI9P8vcFZdz9JGB5ZJNJTR+hihJkiRNpLVyjbUyTM9vHEuTwHI+SZIkSUrBECVJkiRJKRiiJEmSJCkFQ5QkSZIkpWCIkiRJkqQUDFGSJEmSlIIhSpIkSZJSMERJkiRJUgqGKEmSJElKwRAlSZIkSSkYoiRJkiQpBUOUJEmSJKVgiJIkSZKkFAxRkiRJkpSCIUqSJEmSUjBESZIkSVIKhihJkiRJSsEQJUmSJEkpGKIkSZIkKQVDlCRJkiSlYIiSJEmSpBQMUZIkSZKUgiFKkiRJklIwREmSJElSCoYoSZIkSUrBECVJkiRJKRiiJEmSJCkFQ5QkSZIkpWCIkiRJkqQUDFGSJEmSlIIhSpIkSZJSMERJkiRJUgqGKEmSJElKwRAlSZIkSSkYoiRJkiQpBUOUJEmSJKVgiJIkSZKkFAxRkiRJkpSCIUqSJEmSUjBESZIkSVIKhihJkiRJSsEQJUmSJEkpGKIkSZIkKQVDlCRJkibSVC7Lobk81dIah+byTOV8dNVkmBr3ACRJkqRuU7ks80U4d9fnqZVLZPMF5i+6jKW1LJVqbdzD0wFnnJckSdLuqtehVoVqBSolKK/C2jKsLpFZeZDM8v3MFuqcu/uL1MolAGrlEufu/iKz07kxD15yJkqSJEmbqdebX7XOr1qNTL228V6t8/3M+nn13s9R3/K22eNH1gNUS61cIpsZ1Q8qDc4QJUmStF/U60C9J9A0Zn7agk1H6Kl3hZ7ezw6aW+pkIJOBbBYybV/ZPPVstvFeJkt9/fVs5/dtn61lp8jmCx1BKpsvUNs6f0kjZ4iSJEkah1bgSZqtqdfI1LpCUFvA6Q1DG58fPPDQFXSaf+amGqGmLQi1h5x6T0Bq/35400TLqzXmL7psvaQvmy8wd9FlLK1Wh3YPabsMUZIkSYNoL2nrKV1Lmv1JKHnr+uz2Ak/bTE92inque1YnsxF+ugNPJgvZ5vutWaMJVanWWFrLMnvx5UxNZahU6iytVm0qoYlgiJIkSftP9xqetpmeTcvW+gShRknb4HVk7UFmPeBkc9Qz+Y0g1BVyNmZ/Ej474YFnVCrVGg+eq7G4uMA3Hjg77uFI60YaokIIzwVeApwFnhVjPNX23jcDbwO+I8Z4fJTjkCRJEyypcUFi2VrnOZmeILRR8pYq8NC9hqf1/VRbGVu2q8Qt01Py1vH5Axh4pINkZCEqhLAIvAK4ErgKeBPw1LZT7m2+/9FRjUGSJA3ZJp3aqNf7NC5oK2lLmOnJ1FMGnqQZm+xU16xO+3m9jQw6Z3sMPJLSGeVM1JOA22KMyyGEDwO3hBAyMcY6QIzxAeC2EMIIhyBJ0gGW2KmtbbamJwgNWPI26O0hofFAs1Nb27qeelfpWr075IyocYEkbdcoQ9QJIALEGGshhPuAY8CZEd5TkqS9a7PZmq521Mklb72f3VmntsxGp7aeWZ3eNT29ocfAI2l/GnVjiWzb8QKkKFDewvHj88O6VCqLiwtjua8kabLU6zXq1Rr1Wg1qVeq1KvVa67Xmca1KvVqFWm3j+/XXG3/S9jopytrIZslkc2SyWTK5XPO4+X02B7lsx/cb5zUaHLR/n8nmIJMhY+jRBPMZTJNklCHqHuCxACGEBeAocN+wLn7mzBK1Xd5tbXFxgdOn7QwjSXtOxxqeesJaneQGBcmd2prv7aRT2/pszRT1qeT9djZmfxI+m7ZxQR2oNr+SX5Am2jiewbLZzNh+aa/JN8oQ9RHgdSGEWeBq4IPA00MID40x3jTC+0qS9rKBOrUllK11BKHOkrftd2prCzC5fp3aMn0aF9ipTZL2q5GFqBjj10MINwK30mxxTqM73yUAIYSfB54CLIQQbgd+Mcb430Y1HknSCCQ2LmibrenTqW29i9vQOrX17qtT725FndSpLXGmx8AjSdpcpp6m/noyXALcaTmfJKXUCjxt7ag7Zmu6OrAlzgAlzPRsq3FB12xNe6e2njCUtPGogUc6UMZczncpcGpXb66JN+rGEpKk7eoKMr2d2pL36skkhKCdd2rL9Ja0ZdsCUCsM9e3UlgEsa5Mk7Q+GKEkahoS20+vNCfrtt1Nrlq4lblo6eOCBhMYFzQ5s9Vx+Y8Ymm910TU/HZw08kiT1ZYiSdPCsB5fN9tvpLXnr7dTWtmnpsDq1tYecnk5tfUrebFwgSdKuMkRJmmxJndraAk6/xgUbsz+9n03duCCpU1t2qjfktDUoSO7Ultn4U5Ik7VmGKEnD069TW7MddaZPENq05K1eT7+Op3u2JptvBp7kVtT17pBj4wJJkrQJQ5R0UHV0akuYrUnYeDS55K3zs9tvXNC1F0/SpqOZTG/jAgOPJEnaZYYoaa9IbFzQVp6W0MEtsXFBbaeBp222JjtFPbdJp7akkNP63sYFkiRpjzJESaPQE1zamhNsVrbWJwg1Stp22rggRz2T71jX09u4oHNdj53aJEmSehmipKTGBT1la70zPZmeILTNTm10d1zLJHdq65jVyfRpXGCnNkmSpFEzRGlv6dupras5QWKntuSSt9Sd2hJna7o7tXU2KOgNPJnO0CNJkqQ9wxCl0Uns1NY2W9MnCPUveatvfx1P+2xNNt8IOT2d2bJtm5Fmkj9r4JEkSTrwDFHasNlsTa0rBA1a8jborSGh41pmo1Nbz6xOpmdNT89nM9kR/sOSJEnSQWWIGsBULsvsdI5qaY1Dc3mWV6tUqrXxDqpvp7Z6wlqdUXVq65qtyeao5/Jtr2U2XdNj4wJJkiTtRYaoLUzlsswX4dxdn6dWLpHNF5i/6DKW1rKDB6mO4NJnv52EmZ7eTm0bMz3D6NTW07hgkE5tNi6QJEnSAWeI2sLsdG49QAHUyiXO3f1FZh/2TZz92uk+a3g6Z4m236mtLcDkkjq1ZRJCj4FHkiRJGiVD1BayGdYDVEutXCJLncy5+zo3D21vTtC2Lid5TU+/mR4DjyRJkjTJDFFbqNUhmy90BKlsvkAtO0XtxOVjHJkkSZKkcbB92RaWV6vMXXQZ2XwBaASouYsuY3l1zI0lJEmSJI2FM1FbqFRrLK1lmb34cqamMlQqdZYmoTufJEmSpLFwJmoAlWqNB8+VyRWKPHiubICSJEmSDjBDlCRJkiSlYIiSJEmSpBQMUZIkSZKUgiFKkiRJklIwREmSJElSCoYoSZIkSUrBECVJkiRJKRiiJEmSJCkFQ5QkSZIkpTA17gFsQw4gm82M5ebjuq8kSdJBttvPYG33y+3qjbUn7MUQdQLg6NG5sdz8+PH5sdxXkiTpIBvjM9gJ4IvjurkmU6Zer497DGkVgccAXwGqYx6LJEmS9qccjQD1KWBtzGPRhNmLIUqSJEmSxsbGEpIkSZKUgiFKkiRJklIwREmSJElSCoYoSZIkSUrBECVJkiRJKRiiJEmSJCkFQ5QkSZIkpWCI6hJCuDaEUEz5maeEEM4f1ZgkSZL2o7TPXSGEXw4hfOsoxyQNwhDV60XAdMrP/FvgghGMRZIkaT9L9dwVY3x5jPEzIxyPNJBMvV4f9xgmRgjhTcALgQj8OjADPBsoAy+KMX4ihPAy4KeA+4Drge8E/iPwZeD9McZXjGHokiRJe0rXc9d/Bx4KHAe+AvwS8GbgUqAEPDXG+JUQwu8Db44xfjyE8Fng/cCPAPcCT44xlnb/J9FB5ExUmxjjy4B7gMcDfwc8AngkcBVwY/O01wKPAZ4AfCbG+Ebg08AzDFCSJEmD6Xru+hhwDfCqGOOLgQeAV8YY/znwYeB5CZe4HPgQcCVQAK4e/ailBkNUf08GvptGQPoTNsr1bgF+A7g8xvjAeIYmSZK07/xJjDECxBjPAvMhhDcA3wtcnHD+UozxkzHGGvA3uLRCu8gQ1V8W+E8xxkc1vx4OEGN8KfBq4C0hhBePdYSSJEn7R7V1EEL418AvAL8FvImtn1nLQGZ0Q5M6GaJ6fRk4SWPq+LoQwhxACOHCEMJUCOG7Yox/A9wEfF/XZyRJkjS4fs9Q3wt8IMZ4O/AduzskaWuGqF63AB8E/jnwO8BtIYTbgKfTaDRxXQjhb2j8duQNzc+8D7g5hPDm3R+uJEnSnnULjeeuq7te/w3gJ0IIHwMquzskaWt255MkSZKkFJyJkiRJkqQUDFGSJEmSlIIhSpIkSZJSMERJkiRJUgqGKEmSJElKwRAlSXtYCOEjIYTzxj0OSZIOEkOUJO1tTxz3ACRJOmjcJ0qS9qgQwruBZ9PYiHIKyMUYayGEWeAU8AjgU8Bv0QhbR4BfjjG+vfn5HwZeDRSAZeCnY4x/scs/hiRJe44zUZK0R8UYf7x5eAL4W+DJze+fCfxxjPF08/tjwGOAq4HXhRC+NYTwzcCNwA/EGB8FXAf8XghhbrfGL0nSXjU17gFIkobiV4HnAR8AfhL4mfb3Yox14EshhA8BTwJWaISvPw4htM6rAQ+nEcgkSVIfhihJ2h/eB9wYQngCMB9j/ETbe5W24yxQBXI0Zque0XojhPAw4J7dGKwkSXuZ5XyStLdVgXyMcRn4TeBdwK91nXMtQAjhIhqzUB8E/hh4UgjhiuZ7PwDcAczs0rglSdqznImSpL3td4GPhxCeCrybxtqm93adc2kI4TYaAenfxxgjQAjhOuC3QwgZGrNVT4kxLu3e0CVJ2pvszidJ+0AzCP0H4OIY4wvaXj8FPC3G+OkxDU2SpH3HmShJ2h/+EfgK8CPjHogkSfudM1GSJEmSlIKNJSRJkiQpBUOUJEmSJKVgiJIkSZKkFAxRkiRJkpSCIUqSJEmSUjBESZIkSVIK/z9B1pukOgRjawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAF2CAYAAABtbwl0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5Rmd1kn+m910kkMKbTTFNJhHMOIeXRkMhPAyWDUJKgRueRg4oRZzQhZc+I4DoyRy8FwvCC3qDlDIzOe6AyHcDuwGOXkoA5kEhRHueYwCRJEfFgCzXFoNUkboDMhmKTq/PG+zam0lfTb2bX7rar+fNaq1e+79++391Ordzr1rd/lXVhZWQkAAAAP3bZ5FwAAALDZCVYAAAADCVYAAAADCVYAAAADCVYAAAADHT/vAtbJiUm+M8lfJLlvzrUAALB1HZdkV5KPJvnqnGthA9kqweo7k7x/3kUAAHDM+J4kH5h3EWwcWyVY/UWS3HHH/8jyss/lAgBgHNu2LWTHjocl058/4aCtEqzuS5Ll5RXBCgCAo8HyE+7H5hUAAAADCVYAAAADCVYAAAADCVYAAAADjbp5RVVdluTyJAeS7O7uvWu02ZPkrO4+f9Y+AAAAG8loI1ZVtZTkiiRnJ3llkj1rtPm2JN99JH0AAAA2mjGnAl6Q5KbuvivJ9UnOqaqFQ9q8JskvHWEfAACADWXMYLUrSSdJdy8nuSPJqQdPVtXTk/x5kptm7QMAALARjf0BwauD22KSlSSpqhOS/EySpyc5ZZY+s9i589BLAQAAjG/MYLUvk7VSqarFJDsyGYFKJuuqdib5z0lOTPItVfXaJB99kD6HtX//nVlenjmHAQDAEdm2bcEv81nTwsrKOEGkqh6R5CNJzkzyfUkuTfIbSU7r7teuavfNSd7U3eev1ae7L57hdqcn+ZxgBQDAmFYFq8ck2TvfathIRltj1d23J7kyyY1JXprkRUkenclDeCR9AAAANrTRRqyOstNjxAoAgJEZseKBjL15BQBsaYsPPyknnbh93mUwg7u/ek8OfPnueZcBbFGCFQAMcNKJ27P7JW+bdxnM4O1XPTsHIlgB4xjzc6wAAACOCYIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQIIVAADAQMePefGquizJ5UkOJNnd3XtXnXt+kucmeUSSt3b3z0+P/3mS26fN3t/dPzlmjQAAAEONFqyqainJFUnOTHJukj1JLlrV5JYkZyfZnuQzVfX67v7zJAe6+6yx6gIAAFhvY04FvCDJTd19V5Lrk5xTVQsHT3b3HyZZSPKPknwxyb6q2p7kqyPWBAAAsO7GDFa7knSSdPdykjuSnHpIm/ck+WCSf93d9yX5+iS7quoDVfWhqnrSiPUBAACsi1HXWOX+wW0xycrqk939g1V1RpJrq+rJSW5L8swkNye5OMlbkzx21pvt3HnK4IIBgK1raWlx3iUAW9SYwWpfJmuoUlWLSXZkMmp1P9396ap6b5ILuvv/TPKRaZ93JLm6qrZ39z2z3HD//juzvLxy+IYAsE78oL653HbbgXmXwCa3bduCX+azpjGD1Q1JXlFVJyc5L8l1SS6pqtOSvDnJZUlek+TkTNZjXVtV5yf5bHd/Psn5SfbOGqoAAADmZbQ1Vt19e5Irk9yY5KVJXpTk0Ukek8lmFQ9L8tEkn0jyG939/iS3JvkPVfXHSf63TLZjBwAA2NBGXWPV3dckuWbVoT2rXr9s+rW6/SeTPGXMmgAAANbbmLsCAgAAHBMEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIEEKwAAgIGOH/PiVXVZksuTHEiyu7v3rjr3/CTPTfKIJG/t7p8/XB8AAICNaLRgVVVLSa5IcmaSc5PsSXLRqia3JDk7yfYkn6mq1ye5+zB9AAAANpwxpwJekOSm7r4ryfVJzqmqhYMnu/sPkywk+UdJvphk3+H6AAAAbERjBqtdSTpJuns5yR1JTj2kzXuSfDDJv+7u+2bsAwAAsKGMusYq9w9ui0lWVp/s7h+sqjOSXFtVT56lz4PZufOUh1onAHAMWFpanHcJwBY1ZrDal8kaqlTVYpIdmYxA3U93f7qq3pvJNMCZ+jyQ/fvvzPLyzDkMAAbzg/rmctttB+ZdApvctm0LfpnPmsYMVjckeUVVnZzkvCTXJbmkqk5L8uYklyV5TZKTMwlV1yb51KF9ultSAgAANrTRglV3315VVya5MdOt0zPZ4e/0TDareFiSj2ayhuqN3f3+JFmjDwAAwIa2sLKyJQaETk/yOVMBATjalpYWs/slb5t3Gczg7Vc921RABls1FfAxSfbOtxo2kjF3BQQAADgmCFYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADCVYAAAADHT/mxavqsiSXJzmQZHd371117sIkP51kZ5K3dverp8c/nOTkJMtJPtPdPzJmjQAAAEONFqyqainJFUnOTHJukj1JLlrV5NuSfH+SlSRdVb/Z3Z9OspjkzO5eHqs2AACA9TTmiNUFSW7q7ruq6vokb6qqhe5eSZLuvupgw6q6Ock3Jfn09JxQBWw4iw8/KSeduH3eZTCDu796Tw58+e55lwHAMWTMYLUrSSeToFRVdyQ5Ncn+1Y2q6vhMRrU+MT30iKp6b5JHJPn57v6dEWsEmNlJJ27P7pe8bd5lMIO3X/XsHIhgBcDRM+oaq9x/c4zFTKb9HepfJXlfd986ff+MJH+c5AlJ3l1Vj+rur8xys507TxlSKwBbyNLS4rxLYAPyXABjGTNY7UtydpJU1WKSHUnuWN2gqi5I8twk5x881t0fnb78QFXtTfLoJH82yw33778zy8trZTeA4fxAtrncdtuBo3Ifz8XmcrSeC7aubdsW/DKfNY253foNSc6qqpOTnJfkuiSXVNULkqSqnpjk6iQXd/ed02OPr6ozp68fl+QbkuwdsUYAAIDBRhux6u7bq+rKJDdmut16JrsCnj5t8p4kdyW5tqqOS/K7SV6X5HVV9S1JTkxyaXffO1aNAAAA62HUNVbdfU2Sa1Yd2rPq3CMfoNvFY9YEAACw3sacCggAAHBMEKwAAAAGEqwAAAAGEqwAAAAGEqwAAAAGmmlXwKp6ZpJfyeRDfhemXyvd/fARawMAANgUZt1u/ZeTvDDJzUlWxisHAABg85k1WH2xu68dtRIAAIBNatY1VjdW1Q+NWgkAAMAmNeuI1VOTPL+q/ibJ38QaKwAAgK+ZNVh936hVAAAAbGIzTQXs7s8nOTvJLyR5dZJzp8cAAACOeTMFq6p6cZL/NcnHM9kZ8IVV9bNjFgYAALBZzDoV8DlJvru7v5wkVfWGJB9J8qqxCgMAANgsZt0VMAdD1fT1l5LcM0pFAAAAm8ysI1Z7q+ryJFdP3z8vyf87TkkAAACby6wjVj+R5IeT3DX9ujiTcAUAAHDMm2nEqru/kOS8qjo5ybbuvnPcsgAAADaPBw1WVfUr3f1TVfU7SVZWHU+SdPeF45YHAACw8R1uxOr3pn++c+xCAAAANqsHDVbd/TvTl4/t7p9bfa6qXpfkzWMVBgAAsFkcbirgy5PsSPKsqvr6Vae2J/nBJJePWBsAAMCmcLipgDcm+c4ky0n2rzp+b5Jnj1UUAADAZnK4qYDvSfKeqlrp7lccpZoAAAA2lVk/x+qSUasAAADWVVX931V17jpc51+uev2cqvrRdbjmS6rq+4deZ2AND6+qZ616/8aq+p8e6vVm+hyrJF1Vr0/y/iRf+wyr7r72od4YAAC2qr+5596/PGH78d+4jtf7qxO2H/+o9brerKafY/sTSf5jknT3W9bjut191XpcZ6C/k+SfJflP63GxWYPVqdOvx646tpJEsAIAgEOcsP34b9z9kret2/XeftWzZwppVfW8TDaY+3ySb5weu6O7d0xfvzHJu7r7t6rqY0n+jyQ/nuS7krwwyQ8l2ZnkZ7v7N5K8b9KtPpbkeZnsv/AN3f3yqvp7mQSuRyb5kyQ/3t1fmt7jT5I8NckZSf5pd3/okDpfm+SPuvvNVXVLkjcmOTgS9gPdvf+Q9rckeUuSS5P8aZJfT/IL01p/oLv/+5HUk+SWJL+VZGn6vf3w9FbfU1X/Jsm3r1X3g5lpKmB3n9/d5yf5gSQXTN8/edabAAAA46qq78gk/DwxydOS/I/DdPmGJDu6+8zuvjPJu7v7SUkuSPJL0zbPStLdfdYaIePNSV7X3WcmuTnJK1ede0ySJyd5eZLnH6aOxST7uvvxmYSmix+gzeeS/MNMwtEzuvu7Mwl+u4+0nun3e1mS359+b3un7U7OJPPMUvf9zBSsquqRVXVdJn85d1fV+6rqtCO5EQAAMKrzk/xmd3+5u/8myV/O0OfqVa8/V1U/leQVSR79YJ2q6mFJviPJe6aHXp/JxzEddH13r2QScGaZwvhfpn8+WPvf7e77Mhltet/02MeSPGod6znSur9m1s0rfjXJRzIZTnxkJmutfu1IbgQAAIxqeyYjO4d6sJ/570uSqlrMJKzszWRN1V2HudfCjPe55wHaPpBZ2t+zRvv1rudI6545WJ3R3S/v7i929/7uflnuv94KAACYrxuTPKWqTqyqr0/yrdPjn6+qb6+qXUm+5wH6npHk3u5+V5LTk3zd9PitSXZW1f1yw3Qq3acyWZOVTKbV3fBgxVXVs6rqBbN+M1V1VVV97yxtH0o9Sb6Qw4zMHYlZg9X2qjrp4Jvp7iArh+tUVZdV1Seq6kNVdfoh5y6sqg9W1Z9W1c+sOv60qrqlqm6uqifMWB8AABzTpmugfjuTqXJvzWRNUjLZoOK/Jvn3Sd79AN1vSfLZqvqjJP/zwb7d/ZUkv5vkT6rqKYf0eU6SF043lvjHSX4mD+7Rmax1mtUZmcyWm9UR1dPdf5bkS9O88g+P4D5rWlhZOWw+SlX9XCZzFN+YSaD6F5nMcfyFB+mzlOTDSc5Mcm6SH+vui1adf0kmf7krSTqTRWKfS/LpJGcnWUrylu6eJVydnuRz+/ffmeXlw38/AA/F0tJi1nOHJ8bz9quendtuO3BU7uW52DyO5nPB1rVt20J27jwlmQSEvQ/Ubqtst87sZt0V8JVJ3pDJDiFPTfKmTHbKeDAXJLmpu+9Kcn2Sc6rqa/MUu/uq7v5Kd9+dyeKwb8okWd7a3bd29yeTnDAdsgQAgE1jGoIW1utLqNr4Zp0KmEyGFf9TJvvH/+Z0t4wHsyuTkah093KSOzL5LKz7qarjMxnV+sTqPlNfSGL3QQAAYEOb6QOCq+qHk1yTydzL45K8oaou6e7fP0zX1cFtMWuvy/pXSd7X3bdW1ax91jQdlgWALC2ttTEWxzrPBTCWmYJVklcn+d7u/kSSVNXjM1kE9/gH6bMvk7VSB7dv3JHJqNXXVNUFSZ6byZ77B/vU6ibTYzOxxgoYkx/INpejucaKzcMaK4ZatcYK7mfWqYB3HQxVSdLdN+fwI0k3JDlruoPgeUmuS3LJwS0Wq+qJmXwg2cXT7RGT5P/JZDvHb6yqf5Dk8909ywebAQAAzM2sI1bXVdVPZ/JBwfdlspXhH1fVjiQL3f3Xh3bo7tur6spM9tM/kGR3kosy2cEvmXwq8l1Jrq2q4zLZZfB/qarnJXlvknsz2X8eAABgQ5s1WF2RydqqXzzk+I9mMnJ13FqduvuaTNZmHbRn1bk196Tv7usyGd0CAADYFGYKVt29fexCAAAANqtZdwXcluTFSX4oyfZM1k9d2d33jlgbAADApjDr5hW/mOTJSV6XyXS+70ryb8cqCgAAYDOZdY3VU5I8sbvvSZKqeneSj49WFQAAwCYy64jVtoOhKkm6+6tJ7nmQ9gAAAMeMWUes/qiqXpvJdusrSZ6f5JbRqgIAANhEZh2xel6SHUk+lOQjSZaS/JuxigIAANhMZh2xeml3XzpmIQAAAJvVrCNWTx+1CgAAgE1s1hGrz1bVDUk+kOTOgwe7e88oVQEAAGwiswarv57++YQk9yX54jjlAAAAbD6zBqtfTvLWJGdN338wyY+OUhEAAMAmM+saqzcmeX2Sr0tycpJ3JnnDWEUBAABsJrOOWJ3c3f9x1ft/X1U/NkZBAAAAm82sI1Z/WlXfdfBNVT0uyefGKQkAAGBzmXXE6puT/EFVfTzJvZmstfrLqrolSbr7zJHqAwAA2PBmDVY/PWoVAABbyOLDT8pJJ26fdxnM4O6v3pMDX7573mWwBcwUrLr7D8YuBABgqzjpxO3Z/ZK3zbsMZvD2q56dAxGsGG7WNVYAAAA8AMEKAABgIMEKAABgIMEKAABgIMEKAABgIMEKAABgIMEKAABgIMEKAABgIMEKAABgIMEKAABgIMEKAABgIMEKAABgoOPHvHhVXZbk8iQHkuzu7r2rzn1rkquTPL67d646/udJbp++fX93/+SYNQIAAAw1WrCqqqUkVyQ5M8m5SfYkuWhVk1un53/3kK4HuvusseoCAABYb2NOBbwgyU3dfVeS65OcU1ULB09295e6+6bVHapqe5KvjlgTAADAuhszWO1K0knS3ctJ7khy6mH6fH2SXVX1gar6UFU9acT6AAAA1sWoa6xy/+C2mGTlMO33J3lmkpuTXJzkrUkeO+vNdu485UjrA2CLWlpanHcJbECeC9biuWA9jBms9iU5O0mqajHJjkxGrR5Qd68k+ci0zzuSXF1V27v7nlluuH//nVlePlx2A3ho/I93c7nttgNH5T6ei83Fc8FajuS52LZtwS/zWdOYUwFvSHJWVZ2c5Lwk1yW5pKpe8EAdqur8qvrm6dvzk+ydNVQBAADMy2gjVt19e1VdmeTGTLdbz2RXwNOTpKpenuTCJItV9bEkr07yqST/oar+TiabWDx3rPoAAADWy6hrrLr7miTXrDq0Z9W5lyV52RrdnjJmTQAAAOttzKmAAAAAxwTBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYCDBCgAAYKDjx7x4VV2W5PIkB5Ls7u69q859a5Krkzy+u3fO0gcAAGAjGm3EqqqWklyR5Owkr0yy55Amt07PbzuCPgAAABvOmFMBL0hyU3ffleT6JOdU1cLBk939pe6+6Uj6AAAAbERjTgXclaSTpLuXq+qOJKcm2b/Ofb5m585TBhUMwNaxtLQ47xLYgDwXrMVzwXoYdY1V7j8itphkZaQ+SZL9++/M8vLMzQGOiP/xbi633XbgqNzHc7G5eC5Yy5E8F9u2LfhlPmsacyrgviSVJFW1mGRHkjtG6AMAADBXYwarG5KcVVUnJzkvyXVJLqmqFxxJn+42BAUAAGxoo00F7O7bq+rKJDdmunV6kouSnJ4kVfXyJBcmWayqjyV5dXe/c40+AAAAG9qoa6y6+5ok16w6tGfVuZcledkMfUa1+PCTctKJ24/W7Rjg7q/ekwNfvnveZQAAwN8y9uYVG95JJ27P7pe8bd5lMIO3X/XsHIhgBQDAxjPmGisAAIBjgmAFAAAwkGAFAAAwkGAFAAAw0DG/eQWsxW6Rm4fdIgGAjUCwgjXYLXLzsFskALARmAoIAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAwkGAFAAAw0PFjXryqLktyeZIDSXZ3995V53YmeUeSXUne0d2vmh7/cJKTkywn+Ux3/8iYNQIAAAw1WrCqqqUkVyQ5M8m5SfYkuWhVk59L8q4kv5bkg1X12919S5LFJGd29/JYtQEAAKynMacCXpDkpu6+K8n1Sc6pqoVV55+W5H3TAPXO6fskiVAFAABsJmMGq11JOvlaULojyamrzj8yyWemr7+Q5LTp60dU1Xur6mNV9YwR6wMAAFgXo66xyv2D22KSlUPOL6xx7hlJ/jjJE5K8u6oe1d1fmeVmO3eeMqBUNoOlpcV5l8AG5LlgLZ4L1uK5YC2eC9bDmMFqX5Kzk6SqFpPsyGTU6qC/SvLYJJ9MUtP26e6PTs9/oKr2Jnl0kj+b5Yb799+Z5eVDs9uD8x/S5nLbbQeOyn08F5uL54K1eC5Yi+eCtRzJc7Ft24Jf5rOmMacC3pDkrKo6Ocl5Sa5LcklVvWB6/j8nOb+qtmWyucW7q+rxVXVmklTV45J8Q5K9I9YIAAAw2GjBqrtvT3JlkhuTvDTJizIZfXrMtMmrklyY5JYk7+ruTyS5NcnLquqPkvxmkku7+96xagQAAFgPo66x6u5rklyz6tCeVef+OpOdA1e3/+9JLh6zJgAAgPU25lRAAACAY4JgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMJBgBQAAMNDxY168qi5LcnmSA0l2d/feVed2JnlHkl1J3tHdr5oef1qSX0xyb5If6+6bxqwRAABgqNGCVVUtJbkiyZlJzk2yJ8lFq5r8XJJ3Jfm1JB+sqt9O8qkkv5rk7CRLSd6S5Alj1QgAALAexpwKeEGSm7r7riTXJzmnqhZWnX9akvd193KSd07f/+Mkt3b3rd39ySQnVNWuEWsEAAAYbMypgLuSdJJ093JV3ZHk1CT7p+cfmeQz09dfSHLO6j6rjp+W5C8Oc6/jkmTbtoXDNFvbI3Y87CH14+h7qH/HD4XnYvPwXLAWzwVr8VywliN5Lla1PW6UYti0Rl1jlfuPiC0mWTnk/MIa5w7XZy27kmTHQ/wH7N+99JkPqR9H386dpxy1e3kuNg/PBWvxXLAWzwVreYjPxa78/4MEMGqw2pfJWqlU1WKSHUnuWHX+r5I8Nsknk9S0/b7p64MOHj+cjyb5nkxGtu4bWjgAADyA4zIJVR+ddyFsLAsrK7MMCB25qnpEko9ksnnF9yW5NMlvJDmtu19bVXuSfDbJ1UluTPIvMtm84tNJnpTJVME3dbfNKwAAgA1ttM0ruvv2JFdmEppemuRFSR6d5DHTJq9KcmGSW5K8q7s/0d33JnlekvcmeXOSHxurPgAAgPUy2ogVAADAsWLM7dYBAACOCYIVAADAQIIVAADAQIIVAADAQIIVAADAQGN+QDBzUFWXJbk8yYEku7t773wrYqOoqp9O8lNJfqm7Xzfvepivqjolk88R/AdJvpLkn3f3Z+dbFfNWVX83ya8m+XuZPBfP6e5PzbcqNoqqOi2Tzxt9Wnf/wbzrgY3GiNUWUlVLSa5IcnaSVybZM9+K2GB+e/oFSfK4JG/r7rOSvC2TfzPg3iQv7u7HZfJ5ki+ecz1sLL+U5M/mXQRsVILV1nJBkpu6+64k1yc5p6oW5lwTG8T0t8775l0HG0N3f6S7r5++/UCSvzvPetgYuntfd396+ou6M5LcNO+a2Biq6p8kOSXJx+ZdC2xUgtXWsitJJ0l3Lye5I8mpc60I2AzOTnLzvItgY6iq783klzBnJPn1OZfDBjD9Je0vJ3nRvGuBjUyw2npW/50uJlmZVyHAxldVi0lemMS6O5Ik3f2HSU5O8uEkV825HDaG5yb5/e7+3LwLgY1MsNpa9iWp5Gs/LO3IZNQK4G+pqhOS/F9JrrRxBat19z1JfiXJM+ddCxvCjyR5elV9OMnTklxdVd8x55pgw7Er4NZyQ5JXVNXJSc5Lcl13G7EC/pbp1J43Jflv3f2WOZfDBjHdWfb93d1JLk6yd74VsRF099MPvq6qNyZ5U3d/co4lwYYkWG0h3X17VV2Z5MZMt1ufc0lsEFW1K8l7kjwqyX1V9Yzu/v45l8V8/USSf5bkxqo6uBj9R7r7M3Osifm7MZPRiNOSfCnJpfMtB2DzWFhZMaABAAAwhDVWAAAAAwlWAAAAAwlWAAAAAwlWAAAAAwlWAAAAAwlWAMegqnpTVb143nUAwFYhWAEAAAzkA4IBtoCqenuSm7r7NdP3P5HkyUn2JfknSRaTLCS5rLs/eEjflSRL3X37oe+r6hlJfjbJCUnuSvLi7v7wUfq2AGDTMGIFsDW8Psmlq95fmuSWJKcleVJ3//0kb05yxawXrKpvTXJlkqd291lJ/mWSa6vqYetUMwBsGUasALaG/5rkpKp6YiYjS0tJXpXkjCQ/XlXfkuS8JAeO4Jo/kGRXkt+rqoPHlpM8NsnH1+MKEKcAAAELSURBVKVqANgiBCuALaC7V6rqDUmek+SrSd6Q5KlJXpfkNUl+K8mfJvnnD3CJhSSpqhNWHTsuye9197MOHqiqb8pkeiEAsIqpgABbx5uSXJjknyZ5YyYjTr/T3b+W5L8leWYmYelQtyV54vT17lXHfy/JBVX1bUlSVU/NZHrh141RPABsZoIVwBbR3X+Z5OYkt3T3viS/nuS8qvrE9Phnkjymqg79t/8nk/zvVXVzkm9P8hfT6/1JJuuq3lFVH0/yyiQXdvedR+UbAoBNZGFlZWXeNQAAAGxqRqwAAAAGEqwAAAAGEqwAAAAGEqwAAAAGEqwAAAAGEqwAAAAGEqwAAAAGEqwAAAAG+v8Az6HuKv5QGyUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Step 4: adjust bins with badrate_plot(). To fine tune the bins, we use a new combiner().\n",
    "\n",
    "# we want to check the bins of 'duration.in.month'\n",
    "adj_bin = {'duration.in.month': [9, 12, 18, 33]}\n",
    "\n",
    "c2 = toad.transform.Combiner()\n",
    "c2.set_rules(adj_bin)\n",
    "\n",
    "data_ = pd.concat([data_tr,data_ts],axis = 0)\n",
    "temp_data = c2.transform(data_[['duration.in.month','creditability','type']])\n",
    "\n",
    "# plot shows stability across train and test\n",
    "from toad.plot import badrate_plot, proportion_plot\n",
    "badrate_plot(temp_data, target = 'creditability', x = 'type', by = 'duration.in.month') \n",
    "proportion_plot(temp_data['duration.in.month'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAAF2CAYAAACPsxINAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5jkZ13n/XdVdVf39ExPMpntkEmWHIjky0Gyhg1XfAQhHFfdBV3cJRo0yyGB5aBB8Vkjsj5rcLMaHwLoI6wiBFBE0FVchRgELhJYJZIQdqK4dwAzQUiAYUhCJmG6urt+zx9V3f2rql9116+7q6sP79d19dVVv+Ndncl13Z+67/v7q2RZhiRJkiRpMNVRN0CSJEmSthNDlCRJkiSVYIiSJEmSpBIMUZIkSZJUgiFKkiRJkkoYG3UD1mACeBJwL7Aw4rZIkiRpZ6oBh4DPALMjbou2mO0Yop4EfHLUjZAkSdKu8P3Ap0bdCG0t2zFE3Qtw330P0Wxu7jOuDh7cx7Fjxzf1npIkSbvdKPpg1WqFAwf2QrvvKeUNNURFxOXAlcCDwKUppSO5fS8HXg98s73phSmlzw9w2QWAZjPb9BC1eF9JkiRtrhH2wVw+oh5DC1ERMQNcBZwPPA24Dnh+7pCTgNenlN49rDZIkiRJ0kYbZnW+5wC3pZQeBm4EnhwRldz+k4D7h3h/SZIkSdpwwwxRh4AEkFJqAvcBp+T2nwS8NiL+LiLeEhG1IbZFkiRJkjbEsAtL5EPaNJCfzPrr7d8PA38G/Djw+4Ne+ODBfetu3FrMzEyP5L6SJEm7mX0wbSXDDFH3ABcBRMQ0cIDWaBQAKaW7F19HxB8Bjylz8WPHjm/6AsOZmWmOHn1wU+8pSZK0242iD1atVkb2pb22vmGGqI8AV0fEFHAxcAPwgog4Hfgt4EeBPwSmgGcBbx9iWyRJkiRpQwxtTVRK6ZvANcAtwC8ArwXOAM4B5oFz2/s+D/xdSumDw2qLJEmSJG2USpZtu+cenQ3c5XQ+SZKk3WHE0/nOAY5s6s215Q2zOp8kSZIk7TiGKEmSJG1ZzYUGX/2ne2kuNEbdFGmJIUqSJElbVmN2lh98yo/RmJ0ddVOkJcN+TpQkSZJ2sCzLmJ+fpzE7x+zsLI3ZBrOzjdbrE7nXs42lfY32tqXXJ1qv5xqLxze44pWXMjk1yX3H7gfga/d+g9MOQX1igmqtPuJPrd3OECVJkrSDLMwvdAaU2dnegHOiK+z0Czgncq8bi4GnNwQ1m801t7c+UWdiot7+PbH0es/UHn7y+a9cOu6ll1wJwIdufh+TU4YojZYhSpIkaUgWFhZoNOZozM4ye2IxpMwujbY0TszSaMz1hJbOgDPXeU7Hse1jcvdYWFhYc3vH6+NMtMNMfSnYtH5PTU1x4MDJrW2TrbDTOr4dfCY7Q9DExAQTk3Xq9eXrTExOdISl8fo4lUqlsC3NhQYfuvl9fO3eb/DSS67kHe9/C6cdOpX6xMSaP5+0UQxRkiRpV2g2mzQayyMpjcZykOkOKK1jigNO9whPz9S03DS2+fn5Nbd3bGysI5wsBZGJCSYnJ9l/0nTP9tax7df1certINMKNhPtfXXqk639S+e3w061unWWy1drdSan6px2qPX+tEOnMjk1PdpGSW2GKEmStOmyLGOuMdcxnWxxXcxSwCkalcmNviydXzA1bek6uYAz15hbc3trtVoraEwsh4/lgFNn7/Te5YAyMUF9YrxnVKYn4ExOdJwzsRhm2vtrtdoG/sW3r/rEBDd86g8ZG3cKn7YOQ5QkSbtclmXMzc0vTxPrXhvTtQZmbimw9E5N61g7s0rAWatqtdo5+rIUPlqjKScfOKk3tOSmpXWvvVmcmrYUdibGe0Z4amMGmlGp1uqccdrBTX/YrrQSQ5QkSVvM/Px84SjM3GxualnXyE1+e2OVgLN8/vIxWZatqa2VSqW1tmUpcHROG9t/8v6l1+P17nUxqwScgjU29YlxxsbG+q6jkaTNYIiSJGkFi5XOFtfGdJdsXhptacz1rKdZKeAsT2HLraNp36O5sPGVzuoTdfZN76U+cUrHupil1/XlUZnCUZ56LiDl1tiMjxtoJO0+hihJ0rbRbDaXR1hylc4ajbniUZkTy+GkKOB0VDorqoC2AZXO6kujL8sjNN2VzgqDT34Upl4vnJq2GHiWA07dQCNJm8AQJUlak2az2VsYoKDSWaN7ClphwBlsatrc3MZWOlsMH5OTE8WVzibGOyuadY3C1OvjK47cbKVKZ5KkjWOIkqQdoKjSWdG0seLKZQVT0xaPWXq4Zm/AWU+ls2qt2vs8mXyls31Tnc+Z6S4EUBRwuiqddVdHs9KZJGmjGKIkaYNlWdYqDNDnuTKzJzqrlxU9o6a7Olq/gLN4j0Zj7YUBuiudjU90PmzzpJP3d1Q6y6+dKVp7M1Hw/BkrnUmSdhJDlKQdL1/pbG7xwZkneqeNza5Snrlv+eeuSmeNxhzN5toLA3SWZu6qdHbSdG4kprfS2dLamKKAkysOkL+Hlc4kSSrHECVpUy0sLPR9cGajMVc4KtNR6azRKJyatjyFba5n+4ZUOqvXe0LL3n1THDh4IBdW+ozK5ANOz9S03Bqbep3x+riBRpKkLc4QJe1i+UpnPYv/u9bAFFYu6xp96Vx701sBrTE7x/z82gsDjI+PLY/K5Cud1evs2bOHk08+qTO0rPDgzO5KZ0UhaLw+bmEASZLUwxAlbRFZlhUu8m/Mdk0bK1hj0z360h2Clh+u2VkWer2Vzgorl7UDzvRJ+1oBpT7eU9GsaPF/0dS0fPGAen3cwgCSJGlLMERJBRYrnS2Gj87nyuQekNnoDC39QlDPeprCgNNYc3uLKp21wkfrdb9KZysGnFxxgKJn1lgYQJIk7VaGKG15fSudNRqd62m6Kp11B5zuSmfLgadr2tk6K51VKpWOtS7dz4856eT9ualjnWtnFl8vP5yz+PkzExMTjC8GnMlWYQBJkiRtDnteKm1hfqG4ZHOuOEC/UZnF7fnSzEUP4ZzrWp+zUZXOliqXtQPK9P59HOyqdNYdcLornS0WB+i3xsZKZ5IkSTubIWpAzYUGX/2nexkbr1Ot1UfdnCU9lc66F/+f6F3YX1TRLF/prDPg9FZAW0+ls9boSXF55qm9Uxw45eQVnz/Ts/i/4CGc3YUBDDSSJEnaSIaoATVmZ/nXT/1xPnTz+5icKg5RzWazFUD6PH+m0cgFnBNdU8iKqqMVVDpbqoDW3rZRlc7yAWViYoLJyUlOOmn/UsBZenBmV0Wz7sX/RSM3VjqTJEnSTmKIWkVzoRVe7vnKvQB86c67OPnASRw//hBveN0blwNOo8FcY27N96nVah0lmzsrl9XZt39fZwW0XCGAFQPOZK54QK4stJXOJEmSpLUxRK1icQRq0U9f/joA3vvBt/G48x/bE1oKA07R1LTFNTXtAGSlM0mSJGl7qKy1AtkInQ3cdezYcZrN4bd9cSTqa/d+g5deciXveP9bOO3QqdQnJrbU2ihJkqSdamZmmqNHH9zUe1arFQ4e3AdwDnBkU2+uLc+RqFVUa3Ump+qcdqj1/rRDpzI5NT3aRkmSJEkaGUPUgOoTE9zwqT9kbNzRJ0mSJGk3s1TagKq1Omc88pBT+CRJkqRdzhAlSZIkSSUYoiRJkiSpBEOUJEmSJJVgiJIkSZKkEgxRkiRJklSCIUqSJEmSSjBESZIkSVIJhihJkiRJKsEQJUmSJEklGKIkSZIkqQRDlCRJkiSVYIiSJEmSpBIMUZIkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkSSrBECVJkiRJJRiiJEmSJKkEQ5QkSZIklWCIkiRJkqQSDFGSJEmSVIIhSpIkSZJKMERJkiRJUgmGKEmSJEkqwRAlSZIkSSUYoiRJkiSpBEOUJEmSJJVgiJIkSZKkEgxRkiRJklSCIUqSJEmSSjBESZIkSVIJhihJkiRJKsEQJUmSJEkljA3z4hFxOXAl8CBwaUrpSMEx1wEXpJSePsy2SJIkSdJGGNpIVETMAFcBFwFvAK4rOOYxwFOG1QZJkiRJ2mjDnM73HOC2lNLDwI3AkyOi0nXMG4FfHWIbJEmSJGlDDTNEHQISQEqpCdwHnLK4MyL+DfBPwG1DbIMkSZIkbaihromiM6RNAxlARNSBXwT+DbBvLRc+eHBNp63bzMz0SO4rSZK0m9kH01YyzBB1D631UETENHCA1mgUtNZBHQT+ApgAzo2IN6WUfmbQix87dpxmM9vYFq9iZmaao0cf3NR7SpIk7Xaj6INVq5WRfWmvrW+YIeojwNURMQVcDNwAvCAiTk8pvQk4DyAizgLeVSZASZIkSdKoDG1NVErpm8A1wC3ALwCvBc4AzhnWPSVJkiRp2CpZtrlT4jbA2cBdTueTJEnaHUY8ne8c4Mim3lxb3jCr80mSJEnSjmOIkiRJkqQSDFGSJEmSVIIhSpIkSZJKMERJkiRJUgmGKEmSJEkqwRAlSZIkSSUYoiRJkiSpBEOUJEmSJJVgiJIkSZKkEgxRkiRJklSCIUqSJEmSSjBESZIkSVIJhihJkiRJKsEQJUmSJEklGKIkSZIkqQRDlCRJkiSVYIiSJEmSpBIMUZIkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkSSrBECVJkiRJJRiiJEmSJKkEQ5QkSZIklWCIkiRJkqQSDFGSJEmSVIIhSpIkSZJKMERJkiRJUgmGKEmSJEkqwRAlSZIkSSUYoiRJkiSpBEOUJEmSJJVgiJIkSZKkEgxRkiRJklSCIUqSJEmSSjBESZIkSVIJhihJkiRJKsEQJUmSJEklGKIkSZIkqQRDlCRJkiSVYIiSJEmSpBIMUZIkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkaRuLiD+NiKdtwHVelnt9WUT85AZc8z9FxLPWe511tmF/RFySe399RPzweq45tv5mSZIkSRtrvJKx0JgH4P6v30d1IQOgVh9jLquMsmk7UkRMAa8AfgcgpfSejbhuSunajbjOOv1z4MeA92/UBQ1RkiRJWrMsy1o/zQwWX2cZWbO1j9z+xX3F2+h4f8rBffz2K3+z536v+O3XwPj4CD7p1hIRrwKuBO4GHtHedl9K6UD79fXAB1NKfxYRtwO/C7wc+D7gZ4EfBA4Cr08pfQD4eOu0uB14FfAk4OSU0i9HxKNohatTgc8DL08pPdC+x+eBHwLOA/59Sumvu9r5JuBzKaV3R8Rh4HpgcYTr2SmlY13HHwbeA7wI+D/Afwf+S7utz04pfaVMe4DDwJ8BM+3P9m/bt/r+iPgp4LFF7V6NIUqSJG2Ych3q5tL2pX25TnSWZdD1Psvo3dbs0zHv7rQ3l69ReO3uNi9to+tezRUDQvG16XPt3s9R/Hly9+u+V+H96bnXSvfs/Rz0Dz899x/Ov6WfvObFw7nwDhARj6cVdC4ETgA3rXLKycCBlNL57fM/lFK6OiLOphWePgBcQit0XdA+5km5898NXJtS+vOI+E/AG4Cfbu87B3gGcAXwamClMDIN3JNSemJE/AHwo7RHvrqOuQv4F8DtwHNTSk+JiN8CLgWuLdOelNKlEXE58JqU0r9tfzaAKeDZA7a7hyFKkjQUS98sD9BpXO6srd6hbvbp/C5fh1U6h/071M2s2b72SmGg67p9A8Jg7V6+Dn2uPUiHutmxfbCAwOAd6sK/XfF/q12tApVKZfmnWoGu95XK4jYKti2/X/q9uC/3vlKpQLX3XpVKdYX7s3zM0j0Z7F6VCpWu+9F1THe7ez7P4v0Kr9392Vq/J6enRv1fdCt7OvBHKaVvA0TE1wY4562513dFxGuAJwJnrHRSROwFHg98uL3p7cCnc4fcmFLKIuKztKbMreYv278/C5zW55iPppQW2qNSH29vux143Aa2p2y7OxiiVuF8XGljDNyhLjndo19nsbBD3Wx2fsM60Le5g3WoV+qwF34rXKJDvdhu8p9lxb9RQSe3z99o5U50mTDS2aFuNptD+3Z62+jToa6u2LHOdyjp0xFdoUNdbXWiq9UqlRqrdOK7t63Soe7T0V29Yz9Ah7qa7/xT2EGn0m/b4GGkX+e/47/Vin+jxfedf+/8308bpzo3N+ombGXjtEZsuq1UNG4BICKmaQWT/0IrgDx3lXsV/cMuus9cn2P7GeT4ua7XlSG0p2y7AUPUqhYa87zt5W/u2e583PXp3yFd4/zppc7xcgd04Okeq3Wo1zzdY4AOdXe7S0z3WLVDvcbpHgN1qLvDSN+Oded/q92suCM4QCd3xW+lKdjW21lcrUO9ckc3v41V79Xdoa62O5k9nc+V7tdxLfp2qNcbRlbqUFer1a5rU6LNdqglDd0twBUR8QvAJPDo9va7I+KxwP3A9wMfLDj3PGA+pfTB9rTAPe3t3wAORkQ1pdRcPDildDwi/oHWGqq/AC4HPrJS49qV8E5PKb1pkA8TEdcCf5FSunm1Y9fSHuCrrDLiVpYhao3mZxt84W9T3w71Wqd7DNKhXvt0j9U71K1vjzvDyIbOn3a6B1DQoa726wiu8g3rah3qwukWg0z3GKBDvY7pHqt2qEtM98j/jQbqUBdeZ/V2D9ShXqFjLUkqp1Yfa31pDdRqFRZys4Gau7wrkVL664j4n7SKJiRaa4igVTziE8AngQ/1Of0w8I8R8TlaI1J3ta/5nYj4KPD59lS/vMuAt0fENe37XbFKE88Azi7xkc6jVSRiUKXak1L6YkQ8EBF3AD9R4j59VbZhh/Zs4K5jx47T3IT/g6pzc4UjUT95zYv5vdddv/4brGP+dGsaAeWne/S7V25b/294N2D+9Kod6/x1ys2fLrp20d+u6F6rdqj7Bp1+n6P/KIEkSRrczMw0R48+uKn3rFYrHDy4D+Ac4Mim3lxbniNRazR1YB8//mtXrNqh7tjndA9JkiRp2zNErVG1WmP/zMmjboYkSZKkTWaIWoXzcSVJkiTlrVQGUcBcVqE5Pk5zfJyTH3Fg6bXlzSVJkqTdyRAlSZIkSSUYoiRJkiSpBNdESZIkSdtMRFwOXAk8CFyaUjoy2hbtLo5ESZIkSdtIRMwAVwEXAW8Arhtti3afUiEqIkrV9I6IyyPijoj464g4u2vfqyPiMxFxV0RcXea6kiRJ0nbwrcO3Xvqtw7ce+dbhW5vt35duwGWfA9yWUnoYuBF4ckRY9WwTDRSiouXzwN9HxBkR8Q8R8ZhVzlktIR9u73sM8JKIeGTp1kuSJElbVDswvR04C6i0f799A4LUISABpJSawH3AKeu8pkoYdE3Ub9Kac3ltSumrEfGbwO8AT13hnKWEHBE3Au+KiEpKKQNIKd0cETXge4D7gXvW/CkkSZKkTfatw7deBrxkhUO+F5jo2jYFvPNbh299WZ9z3nnK+Re+Z4Db5wdDpgGfYLqJBp3OdzCl9FeLb1JKbwX2r3LOIAn5w8D/Al6ZUloYsC2SJEnSdtAdoFbbPqh7gACIiGngAK2+tjbJoCNRWURM0k64EXEaUBvgvBUTckrpX0XEecCfRMQzUkrfGLA9HDy4b9BDN9TMzPRI7itJkrSbbcU+WHvEqO+o0bcO33qE1hS+bnefcv6FF6/j1h8Bro6IKeBi4IbF2V7aHIOGqLfRWrR2akT8N+DHgV9b5Zx7aK15WjEhp5TujIi/ojX97/cHbA/Hjh2n2dzcfyszM9McPfrgpt5TkiRptxtFH6xarWzEl/avo7Umaiq37eH29jVLKX0zIq4BbqFd4nw911N5lSwbLIhExFOBf01rBOrG/PS+Psf/M+DTwPnAM4EXAR8ATgfeDVwOvJHWP6pPA/8xpfTJAZpyNnCXIUqSJGl3GHGIOgc4stbrtItIXAOcCXwZeN0p51/4BxvRRo3OQCNREfGGlNJ/Bm7ObXtLSunKfuf0ScjPpxWC7gf2Ap+htU7q+gEDlCRJkrRttAOToWmHWXEkKiJ+mdY0vEuA9+d2jQP/KqX0qOE2r9DZOBIlSZK0a2znkSjtTKuNRN0CPAloAsdy2+eBFw6rUZIkSZK0Va0YolJKHwY+HBE3pJT+dpPaJEmSJElb1qDV+e6LiLcA+2g9bbkGfFdK6clDa5kkSZIkbUGDPmz3D4A68H205oQ+DrhjSG2SJEmSpC1r0BA1nVJ6Ba1nRd0APBv4v4bWKkmSJEl9RcTPR8S9EdG3WraGZ9AQtVhU4ovAd6eU7gd8KrIkSZI0Gv+z/aMRGHRN1Bcj4s20HpL7jojYR6vMuSRJkqQ+bvnV658O/AatmVyPXXx90VUv/tp6rptS+oeIuGcDmqg1GHQk6hXAJ1NKtwNvB54BvGxorZIkSZK2uXaA+gvgMcB7c6//8yjbpfUbdCTqz1NKzwRIKb0NeNvwmiRJkiRtfbf86vWXAS9Z4ZALgT20qltfzPIAxhW3/Or1j+9zzjsvuurF79mwRmooBh2JOjki9g61JZIkSdLOchh4AGiy3O9uAl8YWYu0IQYdiXoIuDsiDgPHFzemlJ43lFZJkiRJW1x7xKjvqFFuOl9+4GIeuOmiq178yiE3T0M0aIh6x1BbIUmSJO08v0HrWasA3wFq7ff/DlhziIqIQ8CHgdOAhYh4bkrpWetsq0oYKESllN497IZIkiRJO8yzaRWR+Pe0QtMzaAWoS9Zz0ZTSvcAF626d1qySZf0f9xQRTfo/DypLKQ06krWRzgbuOnbsOM3m5j6qamZmmqNHH9zUe0qSJO12o+iDVasVDh7cB3AOcGRTb64tb7UQNEOrmsgbgLuB3wYWgBcBZw21ZZIkSZK0Ba0YolJKxwAi4sKU0ityu34jIm4dasskSZIkaQsatMT53oiIxTcR8QRgYjhNkiRJkqSta9A1Ta8HPt0ucV4FHgdcOrRWSZIkSdIWNWh1vj+JiE8BT2lvujml9M3hNUuSJEmStqaBQlRE1IEnA3tpFZp4bkR8V0rpF4fZOEmSJEmdImIf8FbgCbSeP/UTKaV/HG2rdpdBp/O9H3gUcAi4HbgI+MSQ2iRJkiSpv+8G3ptSujEiXkWrkvYLR9ymXWXQEPU9wKOBtwHX0VoX9bZhNUqSJEna7n7lub/0NeARBbu+/vo/v/q0tV43pfTp3NtPAT+21mtpbQatzndvSmkeuBP47pTS3wMnDa9ZkiRJ0rZXFKBW2r4WFwGf3cDraQCDjkQdj4hLgf8NXBER/wfYN7xmSZIkSVvbrzz3ly4DXrLGcz/RZ9c7X//nV79nkGtExDTws8APraUNWrtBR6JeBfwL4G+ADLgJ+PVhNUqSJElSf+3Cb/8DuMaiEptv0JEogKcBr6UVoj4JfGgoLZIkSZK2gfaIUd9Ro1957i9lK5x78VrvGxEV4F3ArSmlgUattLEGHYn6beAdwB5gCvhT4HeH1ShJkiRJfb2CVjGJp0fE7e2fc0fdqN1k0JGoAymlt+fe/2ZEvHQYDZIkSZJ2iK/Tpzrfei6aUnorredEaUQGDVFfjIiLUkq3AETE+cCXhtcsSZIkaXtbTxlzbW0rhqiIuIPWGqhp4FMRcRhYoPXcqM8Pv3mSJEnarWq1CpNj1Y7XD80ujLhV0uojUa/elFZIkiRJObVahT3VjAfuPMLk95y39Hrq3DOZne9br0HaFCuGqJTSTZvVEEmSJO1eWZaRzc3TnJunOTfHSafs4/4vfIWvfup2Jk85iSN/9WmqtSqPPu8sQ5RGrkyJc0mSJGlJ1szI5luhp9lo/56bb4eh1utmY64jHPXbls3Nd1z7q7Uaj3zqEznnB59M+sBHAHj8i3+Y78wXtUTaXIYoSZKkXSRrNjuDTlH4aW/rCT9z82T5/fMDrk+qVqiOj1MdH6NaH6cyPkZ1ss7Y9N7WtvFxKvWxpdfV8THG9+3hpEMHufOPP7p0mWN//0UOXPA4XBWlUTNESZIkbXHZQrPvKE/Pto6Rnlz4WTx2YbAIUqlVqSwGn/bv2uQE1f372tva4acdjHq25bZXarXSn3nvRI1v33mEaq3KE176Ixy94ws8+OWvceqFj9/10/ki4kzg/wMeBXwHuCyl9A+jbdXuYoiSJEkagmxhoWOUp2Nkp9F/W2f4ae3PFpoD3bNSq1Gtj+XCzxhjU3t6tq0YfhaDT7U65L/Qyh6aXWDq3DN59HlnsWf/Xg5c8DhOvfDxfGeXB6i2eeDnUkp3RsSrgZ8DfIbrJjJESZIk0S5ssNBsTVfrF3RWWtPTFZhoDtbZr4zVloNLO+iM7ZtaGv3pDDu9ozz5baMOPhttdj5jdj5jT+71dnT+WU/7Gq2H7n798N03rfvZUSmlewAiYgY4D7htvddUOYYoSZK0bWVZRja/0BV2FtfszC8HohW25Ud+yAYMPrkpbkvrfPZPdEx9q6wyxa06Pk5lbIxKtTLkv5K2gEd0/V63iHgq8LH2z2s26roajCFKkiRtqu5S1ktT2BpdxQv6rOnpDkwMknsq5ILMcripTe3pCD750aDl0aHu/WNUKgYfwflnPe0y4CUrHPJ9wDitf6UVIDv/rKdlwBzw133Oeefhu296z2r3TindHBFTwOuAa2lN6dMmMURJkqRVDbOUdV+VSm/ltvo4Y3v3tIJOVzW35fDTu60yVjP4aBTG278rXb/HC44tLaU0FxFvpjWdzxC1iQxRkiTtUFuxlHX3+p6ekZ7c2qBKrWrw0ZbWHjHqO2qUWws1C0zkfn/98N03XbzW+0bE5cAnU0oJ+FHgyFqvpbUxREmStIVkzWapstUbX8q6FWj6lrJepcjBWkpZSzvVYhGJ9hQ+gInDd9+0Ed8M3AK8NSJOBx4AXrQB11QJhihJktYpW1goV7a6T0jK5ubWXcp6tbLVW7GUtbQLfJ12db6NuFhK6Q7gmRtxLa2NIUqStOtsu1LWS+t/dm4pa2kn24iy5tpaDFGSpG1h3aWsu8JPuVLWnWt2ekpZD/IcH0tZS9KOYYiSJA3NtillnS+CYClrSdIqDFGSpA4dpaxXWNOzqaWs+5St7pziZilrSdLmMERJ0g4w8lLWi+HHUtaSpF3AECVJI7IYfAYtW11YyrodfoZdyrow/FjKWpK0SxmiJKmE7lLWg5St7hd+LGUtSdL2ZIiStJy35A4AABGmSURBVKOtu5R11zmlSlm3y1L3LWVdULbaUtaSJG19hihJW85WKWVdGR+j1l7fYylrSZK0yBAlaUN0lrLOhZ/CUtb5sDPMUtYF1dw6Sll3hiMLG0iSpEEYoqRdbK2lrLunuK25lPViiWpLWUuSpG3EECVtMwOVsu73HJ/8aFDDUtaSJElrYYiSNkHZUtZZVzja0FLW03tXXNNjKWtJkqSVGaKkPkZWynp8jEp9HaWsF1/XrOgmSZI0DIYo7RiWspYkSdJmMERppPKlrAcpW91sbIFS1vnwYylrSZKkXccQpdL6lrLumOa2waWsIVeW2lLWkiRJGh1D1C4xUCnrjqBTNMVtOfwMxFLWkiRJ2oEMUQOo1SpMjlU7Xj80O2Bp6HXImlnHiI2lrCVJkqTRG2qIiojLgSuBB4FLU0pHcvueB/w8cBD4vZTSfx1mW9aqVquwp5rxwJ1HmPye85ZeT517JrPzvfPQRlfKeqwj/HSXsi4OOp3nVOuWspYkSZJWM7QQFREzwFXA+cDTgOuA5+cOeQzwLForYlJE/FFK6c5htWetJseqPHDnEb7yyc8ycdI+7v74Z6jWqpxz+gz33PjprqlvQyxlvUKRA0tZS5IkSZtnmCNRzwFuSyk9HBE3Au+KiEpKKQNIKV27eGBEfBZ4JLDlQtRDswtMxzmce/I0d/7xRwF47At/kK/8zWGoVhjbt4fq+P4Vy1bnR4AsZS1JkiRtb8MMUYeABJBSakbEfcApwLH8QRExRmu06o4htmXNarUK1YV57v7Y3y5tu+8LX+aMZ1xUOJ1PkiRJ0s427MIS+eGWaYqLWf9H4OMppW+UufDBg/vW066BzZ+Y5Zt/9yWqtSpPeOmPcPSOL/Lgl+/l9O89n/0HJjelDZIkSbvdzMz0qJsgLRlmiLoHuAggIqaBA8B9+QMi4jnAfwCeXvbix44dp9ncnJGgqXPP5NHnncWe/Xs5cMFjOfXCx/HgiXkWHnpwU+4vSZK0m83MTHP06Ob2u6rVyqZ9aa/tZ5gLcz4CXBARU8DFwA3ACyLiZwAi4kLgrcCPppSOD7Ed6zY7n3F8ttnxemHBqXySJEnSbjS0EJVS+iZwDXAL8AvAa4EzgHPah3yY1kjYn0TE7RHx68NqiyRJkiRtlEqWbbsRlbOBuzZzOt+iUQwlS5Ik7XYjns53DnBkU2+uLc8625IkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkSSrBECVJkiRJJRiiJEmSJKkEQ5QkSZIklWCIkiRJkqQSDFGSJEmSVIIhSpIkSZJKMERJkiRJUgmGKEmSJEkqwRAlSZIkSSUYoiRJkiSpBEOUJEmSJJVgiJIkSZKkEgxRkiRJklSCIUqSJEmSSjBESZIkSVIJhihJkiRJKsEQJUmSJEklGKIkSZIkqQRDlCRJkiSVYIiSJEmSpBIMUZIkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkSSrBECVJkiRJJRiiJEmSJKkEQ5QkSZIklWCIkiRJkqQSDFGSJEmSVIIhSpIkSZJKMERJkiRJUgmGKEmSJEkqwRAlSZIkSSUYoiRJkiSpBEOUJEmSJJVgiJIkSZKkEgxRkiRJklSCIUqSJEmSSjBESZIkSVIJhihJkiRJKsEQJUmSJEklGKIkSZIkqQRDlCRJkiSVYIiSJEmSpBIMUZIkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkSSrBECVJkiRJJRiiJEmSJKkEQ5QkSZK2pLFalf17x1lozLJ/7zhjNbuu2hrGRt0ASZIkqdtYrcq+CXjo7jtpzjWojtfZd+a5HJ+tMr/QHHXztMsZ5yVJkrS5sgyaC7AwD/MNmDsBsw/DieNUvvNtKg/fz1Q946Evf4nmXAOA5lyDh778JaYmayNuvORIlCRJklaSZe2fZudPs0klay7va3burywdl/WeR7bqbasHT14KUIuacw2qlWF9UGlwhihJkqSdIsuArCfQtEZ+csGmI/RkXaGn99xBc0tGBSoVqFahkvupjpNVq619lSrZ0vZq5/vcuc3qGNXxekeQqo7Xaa6ev6ShM0RJkiSNwmLgKRqtyZpUml0hKBdwesPQ8vmDBx66gk77d22sFWpyQSgfcrKegJR/v3HDRA+faLLvzHOXpvRVx+vsPfNcjp9Y2LB7SGtliJIkSRpEfkpbz9S1otGfgilvXeeuLfDkRnqqY2S17lGdynL46Q48lSpU2/sXR422qPmFJsdnq0yddR5jYxXm5zOOn1iwqIS2BEOUJEnaebrX8ORGelacttYnCLWmtA0+jywfZJYCTrVGVhlfDkJdIWd59Kfg3C0eeIZlfqHJtx9qMjMzzbceeHDUzZGWDDVERcTlwJXAg8ClKaUjuX2PBt4KPDGldHCY7ZAkSVtYUeGCwmlrncdUeoLQ8pS3UoGH7jU8i+/HctPYql1T3Co9U946zt+FgUfaTYYWoiJiBrgKOB94GnAd8PzcId9o7//osNogSZI22AqV2siyPoULclPaCkZ6KlnJwFM0YlMd6xrVyR/XW8igc7THwCOpnGGORD0HuC2l9HBE3Ai8KyIqKaUMIKX0AHBbRAyxCZIk7WKFldpyozU9QWjAKW+D3h4KCg+0K7Xl1vVkXVPXsu6QM6TCBZK0VsMMUYeABJBSakbEfcApwLEh3lOSpO1rpdGarnLUxVPees9dX6W2ynKltp5Rnd41Pb2hx8AjaWcadmGJau71NJSYoLyKgwf3bdSlSpmZmR7JfSVJW0uWNckWmmTNJjQXyJoLZM3Fbe3XzQWyhQVoNpffL21v/Sa3nRLT2qhWqVRrVKpVKrVa+3X7fbUGtWrH++XjWgUO8u8r1RpUKlQMPdrC7INpKxlmiLoHuAggIqaBA8B9G3XxY8eO09zkp63NzExz9KiVYSRp2+lYw5MVrNUpLlBQXKmtvW89ldqWRmvGyMaKn7ezPPpTcG7ZwgUZsND+Kd4gbWmj6INVq5WRfWmvrW+YIeojwNURMQVcDNwAvCAiTk8pvWmI95UkbWcDVWormLbWEYQ6p7ytvVJbLsDU+lVqq/QpXGClNknaqYYWolJK34yIa4BbaJc4p1Wd72yAiPhl4HnAdETcDvzXlNIfD6s9kqQhKCxckBut6VOpbamK24ZVaut9rk7WXYq6qFJb4UiPgUeStLJKVmb+9dZwNnCX0/kkqaTFwJMrR90xWtNVga1wBKhgpGdNhQu6Rmvyldp6wlDRg0cNPNKuMuLpfOcARzb15tryhl1YQpK0Vl1BprdSW/GzeioFIWj9ldoqvVPaqrkAtBiG+lZqqwBOa5Mk7QyGKEnaCAVlp5eKE/R73k6zPXWt8KGlgwceKChc0K7AltXGl0dsqtUV1/R0nGvgkSSpL0OUpN1nKbis9Lyd3ilvvZXacg8t3ahKbfmQ01Oprc+UNwsXSJK0qQxRkra2okptuYDTr3DB8uhP77mlCxcUVWqrjvWGnFyBguJKbZXl35IkadsyREnaOP0qtbXLUVf6BKEVp7xlWfl1PN2jNdXxduApLkWddYccCxdIkqQVGKKk3aqjUlvBaE3Bg0eLp7x1nrv2wgVdz+IpeuhopdJbuMDAI0mSNpkhStouCgsX5KanFVRwKyxc0Fxv4MmN1lTHyGorVGorCjmL7y1cIEmStilDlDQMPcElV5xgpWlrfYJQa0rbegsX1Mgq4x3renoLF3Su67FSmyRJUi9DlFRUuKBn2lrvSE+lJwitsVIb3RXXKsWV2jpGdSp9ChdYqU2SJGnYDFHaXvpWausqTlBYqa14ylvpSm2FozXdldo6CxT0Bp5KZ+iRJEnStmGI0vAUVmrLjdb0CUL9p7xla1/Hkx+tqY63Qk5PZbZq7mGkleJzDTySJEm7niFKy1YarWl2haBBp7wNemsoqLhWWa7U1jOqU+lZ09NzbqU6xD+WJEmSditD1ADGalWmJmssNGbZv3ech08sML/QHG2j+lZqywrW6gyrUlvXaE21RlYbz22rrLimx8IFkiRJ2o4MUasYq1XZNwEP3X0nzbkG1fE6+848l+Oz1cGDVEdw6fO8nYKRnt5KbcsjPRtRqa2ncMEgldosXCBJkqRdzhC1iqnJ2lKAAmjONXjoy19i6pGP4sGvH+2zhqdzlGjtldpyAaZWVKmtUhB6DDySJEnSMBmiVlGtsBSgFjXnGlTJqDx0X+fDQ/PFCXLrcorX9PQb6THwSJIkSVuZIWoVzQyq4/WOIFUdr9OsjtE8dN4IWyZJkiRpFCxftoqHTyyw98xzqY7XgVaA2nvmuTx8YsSFJSRJkiSNhCNRq5hfaHJ8tsrUWecxNlZhfj7j+FaozidJkiRpJByJGsD8QpNvPzRHrT7Btx+aM0BJkiRJu5ghSpIkSZJKMERJkiRJUgmGKEmSJEkqwRAlSZIkSSUYoiRJkiSpBEOUJEmSJJVgiJIkSZKkEgxRkiRJklSCIUqSJEmSShgbdQPWoAZQrVZGcvNR3VeSJGk32+w+WO5+tU29sbaF7RiiDgEcOLB3JDc/eHDfSO4rSZK0m42wD3YI+NKobq6tqZJl2ajbUNYE8CTgXmBhxG2RJEnSzlSjFaA+A8yOuC3aYrZjiJIkSZKkkbGwhCRJkiSVYIiSJEmSpBIMUZIkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkSSrBECVJkiRJJRiiukTEZRExUfKc50XEI4bVJkmSpJ2obL8rIt4YEU8YZpukQRiier0amCx5zk8Apw2hLZIkSTtZqX5XSum1KaU7htgeaSCVLMtG3YYtIyKuA14FJOB3gT3Ai4A54NUppZsj4meBVwL3AT8DfC/w/wBfBT6YUrpqBE2XJEnaVrr6Xf8DOB04CNwL/BrwZuAcoAE8P6V0b0T8KfDmlNJNEfH3wAeBHwa+AfxASqmx+Z9Eu5EjUTkppZ8F7gG+H/g74PHAdwNPA65pH/ZLwJOApwN3pJT+X+BW4BIDlCRJ0mC6+l2fAF4I/GJK6aeAB4DXpZT+JXAjcEXBJc4D/hI4H6gDFw+/1VKLIaq/HwCeQisgfZzl6XrvAn4POC+l9MBomiZJkrTjfDyllABSSg8C+yLiWuCZwFkFxx9PKX0ypdQEPodLK7SJDFH9VYH/llK6oP3zXQAppdcArwfeEhE/NdIWSpIk7RwLiy8i4keAXwHeB1zH6n3WOaAyvKZJnQxRvb4KnEFr6PhlEbEXICL+eUSMRcT3pZQ+B7wJeFbXOZIkSRpcvz7UM4EPp5RuB564uU2SVmeI6vUu4AbgXwIfAG6LiNuAF9AqNPGyiPgcrW9Hrm2f817gdyLizZvfXEmSpG3rXbT6XRd3bf894KUR8QlgfnObJK3O6nySJEmSVIIjUZIkSZJUgiFKkiRJkkowREmSJElSCYYoSZIkSSrBECVJkiRJJRiiJGkbi4iPRMQ/G3U7JEnaTQxRkrS9PXvUDZAkabfxOVGStE1FxPXAi2g9iHIMqKWUmhExBRwBHg98BngfrbB1MvDGlNLb2uc/F3g9UAceBn4upfQ3m/wxJEnadhyJkqRtKqX04vbLQ8D/Bn6g/f7HgI+llI62358CPAm4GLg6Ip4QEY8GrgF+KKV0AfAy4E8iYu9mtV+SpO1qbNQNkCRtiN8CrgA+DLwc+L/z+1JKGfCViPhL4DnAd2iFr49FxOJxTeC7aAUySZLUhyFKknaG9wLXRMTTgX0ppZtz++Zzr6vAAlCjNVp1yeKOiHgkcM9mNFaSpO3M6XyStL0tAOMppYeB3wfeCfz3rmMuA4iIM2mNQt0AfAx4TkQ8pr3vh4DDwJ5NarckSduWI1GStL39EXBTRDwfuJ7W2qb3dB1zTkTcRisg/XRKKQFExMuAP4yICq3RquellI5vXtMlSdqerM4nSTtAOwj9PHBWSukVue1HgH+XUrp1RE2TJGnHcSRKknaGfwTuBX541A2RJGmncyRKkiRJkkqwsIQkSZIklWCIkiRJkqQSDFGSJEmSVIIhSpIkSZJKMERJkiRJUgmGKEmSJEkq4f8HI6ew5K2XR4wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Assume we want to combine bin #1 and #2...\n",
    "adj_bin = {'duration.in.month': [9, 18,33]}\n",
    "c2.set_rules(adj_bin)\n",
    "\n",
    "temp_data = c2.transform(data_[['duration.in.month','creditability','type']])\n",
    "badrate_plot(temp_data, target = 'creditability', x = 'type', by = 'duration.in.month')\n",
    "\n",
    "adj_bin = {'duration.in.month': [9, 18,33],'foreign.worker': [['no'], ['yes']]}\n",
    "# therefore use this "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 5: WOE transform with adjusted bins \n",
    "combiner.set_rules(adj_bin)\n",
    "\n",
    "binned_data = combiner.transform(selected_data)\n",
    "\n",
    "transer = toad.transform.WOETransformer()\n",
    "data_tr_woe = transer.fit_transform(binned_data, binned_data['creditability'], exclude=['creditability','type'])\n",
    "data_ts_woe = transer.transform(combiner.transform(selected_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>status.of.existing.checking.account</th>\n",
       "      <th>duration.in.month</th>\n",
       "      <th>credit.history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit.amount</th>\n",
       "      <th>savings.account.and.bonds</th>\n",
       "      <th>present.employment.since</th>\n",
       "      <th>personal.status.and.sex</th>\n",
       "      <th>other.debtors.or.guarantors</th>\n",
       "      <th>property</th>\n",
       "      <th>age.in.years</th>\n",
       "      <th>other.installment.plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>foreign.worker</th>\n",
       "      <th>creditability</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>569</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>574</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     status.of.existing.checking.account  duration.in.month  credit.history  \\\n",
       "569                                    3                  3               1   \n",
       "574                                    2                  1               1   \n",
       "993                                    3                  3               1   \n",
       "\n",
       "     purpose  credit.amount  savings.account.and.bonds  \\\n",
       "569        1              2                          3   \n",
       "574        1              0                          3   \n",
       "993        2              2                          3   \n",
       "\n",
       "     present.employment.since  personal.status.and.sex  \\\n",
       "569                         2                        2   \n",
       "574                         1                        1   \n",
       "993                         4                        1   \n",
       "\n",
       "     other.debtors.or.guarantors  property  age.in.years  \\\n",
       "569                            0         2             2   \n",
       "574                            0         2             1   \n",
       "993                            0         1             2   \n",
       "\n",
       "     other.installment.plans  housing  foreign.worker  creditability   type  \n",
       "569                        0        0               1              1  train  \n",
       "574                        0        0               1              0  train  \n",
       "993                        0        0               1              0  train  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "binned_data.head(3) # the values are transformed into \"index + bin\" format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "---\n",
    "### III. Model selection\n",
    "1. **toad.selection.stepwise()**: this function performs feature selection (or model selection formally) with forward / backward / both-way stepwise. The function uses AIC / BIC / KS / AUC as selection criterion. \n",
    "2. The function supports choosing base algorithm (estimator) like 'ols': LinearRegression / 'lr': LogisticRegression / 'lasso': Lasso / 'ridge': Ridge and so on. （toad version >= 0.0.58）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(750, 9)\n",
      "Index(['status.of.existing.checking.account', 'duration.in.month',\n",
      "       'credit.history', 'purpose', 'personal.status.and.sex', 'property',\n",
      "       'age.in.years', 'other.installment.plans', 'creditability'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "final_data = toad.selection.stepwise(data_tr_woe.drop('type',axis=1),target = 'creditability',direction = 'both', criterion = 'aic')\n",
    "\n",
    "final_test = data_ts_woe[final_data.columns]\n",
    "print(final_data.shape)\n",
    "print(final_data.columns) # As shown, 15 features are down to 8 after both-way stepwise "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/zhouxiyu/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='warn', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now ready to model. Fit a lr. \n",
    "Xtr = final_data.drop('creditability',axis=1)\n",
    "Ytr = final_data['creditability']\n",
    "Xts = final_test.drop('creditability',axis=1)\n",
    "Yts = final_test['creditability']\n",
    "\n",
    "lr = LogisticRegression()\n",
    "lr.fit(Xtr, Ytr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "----\n",
    "### IV. Model evaluation and validation\n",
    "1. **Common evaluation metrics**: toad. metrics. KS, F1, AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training error\n",
      "F1: 0.4556701030927835\n",
      "KS: 0.4872853159929582\n",
      "AUC: 0.8107664187464729\n",
      "\n",
      "Test error\n",
      "F1: 0.45079365079365086\n",
      "KS: 0.4771302530763873\n",
      "AUC: 0.7811314913706369\n"
     ]
    }
   ],
   "source": [
    "from toad.metrics import KS, F1, AUC\n",
    "\n",
    "EYtr_proba = lr.predict_proba(Xtr)[:,1]\n",
    "EYtr = lr.predict(Xtr)\n",
    "\n",
    "print('Training error')\n",
    "print('F1:', F1(EYtr_proba,Ytr))\n",
    "print('KS:', KS(EYtr_proba,Ytr))\n",
    "print('AUC:', AUC(EYtr_proba,Ytr))\n",
    "\n",
    "EYts_proba = lr.predict_proba(Xts)[:,1]\n",
    "EYts = lr.predict(Xts)\n",
    "\n",
    "print('\\nTest error')\n",
    "print('F1:', F1(EYts_proba,Yts))\n",
    "print('KS:', KS(EYts_proba,Yts))\n",
    "print('AUC:', AUC(EYts_proba,Yts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------\n",
    "2. **toad.metrics.PSI():** return the PSI of each feature to detect data migration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "purpose                                0.053175\n",
       "duration.in.month                      0.038424\n",
       "age.in.years                           0.017464\n",
       "property                               0.014331\n",
       "credit.history                         0.012744\n",
       "status.of.existing.checking.account    0.001251\n",
       "personal.status.and.sex                0.001096\n",
       "creditability                          0.000545\n",
       "other.installment.plans                0.000047\n",
       "dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "psi = toad.metrics.PSI(final_data,final_test)\n",
    "psi.sort_values(0,ascending=False) # Further tune the unstable feature if any "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "3. **toad.metrics.KS_bucket(**predicted_proba, y_true, bucket=10, method = 'quantile'**)**: output the result table including bad_rate, KS etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>bads</th>\n",
       "      <th>goods</th>\n",
       "      <th>total</th>\n",
       "      <th>bad_rate</th>\n",
       "      <th>good_rate</th>\n",
       "      <th>odds</th>\n",
       "      <th>bad_prop</th>\n",
       "      <th>good_prop</th>\n",
       "      <th>cum_bads</th>\n",
       "      <th>cum_goods</th>\n",
       "      <th>cum_bads_prop</th>\n",
       "      <th>cum_goods_prop</th>\n",
       "      <th>ks</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.012857</td>\n",
       "      <td>0.052979</td>\n",
       "      <td>0</td>\n",
       "      <td>75</td>\n",
       "      <td>75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.143403</td>\n",
       "      <td>0</td>\n",
       "      <td>75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.143403</td>\n",
       "      <td>-0.143403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.053378</td>\n",
       "      <td>0.094267</td>\n",
       "      <td>1</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.013333</td>\n",
       "      <td>0.986667</td>\n",
       "      <td>0.013514</td>\n",
       "      <td>0.004405</td>\n",
       "      <td>0.141491</td>\n",
       "      <td>1</td>\n",
       "      <td>149</td>\n",
       "      <td>0.004405</td>\n",
       "      <td>0.284895</td>\n",
       "      <td>-0.280490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.094706</td>\n",
       "      <td>0.129116</td>\n",
       "      <td>9</td>\n",
       "      <td>66</td>\n",
       "      <td>75</td>\n",
       "      <td>0.120000</td>\n",
       "      <td>0.880000</td>\n",
       "      <td>0.136364</td>\n",
       "      <td>0.039648</td>\n",
       "      <td>0.126195</td>\n",
       "      <td>10</td>\n",
       "      <td>215</td>\n",
       "      <td>0.044053</td>\n",
       "      <td>0.411090</td>\n",
       "      <td>-0.367037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.130818</td>\n",
       "      <td>0.173092</td>\n",
       "      <td>14</td>\n",
       "      <td>61</td>\n",
       "      <td>75</td>\n",
       "      <td>0.186667</td>\n",
       "      <td>0.813333</td>\n",
       "      <td>0.229508</td>\n",
       "      <td>0.061674</td>\n",
       "      <td>0.116635</td>\n",
       "      <td>24</td>\n",
       "      <td>276</td>\n",
       "      <td>0.105727</td>\n",
       "      <td>0.527725</td>\n",
       "      <td>-0.421998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.173933</td>\n",
       "      <td>0.241864</td>\n",
       "      <td>20</td>\n",
       "      <td>55</td>\n",
       "      <td>75</td>\n",
       "      <td>0.266667</td>\n",
       "      <td>0.733333</td>\n",
       "      <td>0.363636</td>\n",
       "      <td>0.088106</td>\n",
       "      <td>0.105163</td>\n",
       "      <td>44</td>\n",
       "      <td>331</td>\n",
       "      <td>0.193833</td>\n",
       "      <td>0.632887</td>\n",
       "      <td>-0.439055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.242644</td>\n",
       "      <td>0.324243</td>\n",
       "      <td>18</td>\n",
       "      <td>57</td>\n",
       "      <td>75</td>\n",
       "      <td>0.240000</td>\n",
       "      <td>0.760000</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>0.079295</td>\n",
       "      <td>0.108987</td>\n",
       "      <td>62</td>\n",
       "      <td>388</td>\n",
       "      <td>0.273128</td>\n",
       "      <td>0.741874</td>\n",
       "      <td>-0.468746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.324983</td>\n",
       "      <td>0.422900</td>\n",
       "      <td>29</td>\n",
       "      <td>46</td>\n",
       "      <td>75</td>\n",
       "      <td>0.386667</td>\n",
       "      <td>0.613333</td>\n",
       "      <td>0.630435</td>\n",
       "      <td>0.127753</td>\n",
       "      <td>0.087954</td>\n",
       "      <td>91</td>\n",
       "      <td>434</td>\n",
       "      <td>0.400881</td>\n",
       "      <td>0.829828</td>\n",
       "      <td>-0.428947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.424597</td>\n",
       "      <td>0.529979</td>\n",
       "      <td>37</td>\n",
       "      <td>38</td>\n",
       "      <td>75</td>\n",
       "      <td>0.493333</td>\n",
       "      <td>0.506667</td>\n",
       "      <td>0.973684</td>\n",
       "      <td>0.162996</td>\n",
       "      <td>0.072658</td>\n",
       "      <td>128</td>\n",
       "      <td>472</td>\n",
       "      <td>0.563877</td>\n",
       "      <td>0.902486</td>\n",
       "      <td>-0.338609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.532320</td>\n",
       "      <td>0.649534</td>\n",
       "      <td>45</td>\n",
       "      <td>30</td>\n",
       "      <td>75</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>0.198238</td>\n",
       "      <td>0.057361</td>\n",
       "      <td>173</td>\n",
       "      <td>502</td>\n",
       "      <td>0.762115</td>\n",
       "      <td>0.959847</td>\n",
       "      <td>-0.197732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.652615</td>\n",
       "      <td>0.921777</td>\n",
       "      <td>54</td>\n",
       "      <td>21</td>\n",
       "      <td>75</td>\n",
       "      <td>0.720000</td>\n",
       "      <td>0.280000</td>\n",
       "      <td>2.571429</td>\n",
       "      <td>0.237885</td>\n",
       "      <td>0.040153</td>\n",
       "      <td>227</td>\n",
       "      <td>523</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        min       max  bads  goods  total  bad_rate  good_rate      odds  \\\n",
       "0  0.012857  0.052979     0     75     75  0.000000   1.000000  0.000000   \n",
       "1  0.053378  0.094267     1     74     75  0.013333   0.986667  0.013514   \n",
       "2  0.094706  0.129116     9     66     75  0.120000   0.880000  0.136364   \n",
       "3  0.130818  0.173092    14     61     75  0.186667   0.813333  0.229508   \n",
       "4  0.173933  0.241864    20     55     75  0.266667   0.733333  0.363636   \n",
       "5  0.242644  0.324243    18     57     75  0.240000   0.760000  0.315789   \n",
       "6  0.324983  0.422900    29     46     75  0.386667   0.613333  0.630435   \n",
       "7  0.424597  0.529979    37     38     75  0.493333   0.506667  0.973684   \n",
       "8  0.532320  0.649534    45     30     75  0.600000   0.400000  1.500000   \n",
       "9  0.652615  0.921777    54     21     75  0.720000   0.280000  2.571429   \n",
       "\n",
       "   bad_prop  good_prop  cum_bads  cum_goods  cum_bads_prop  cum_goods_prop  \\\n",
       "0  0.000000   0.143403         0         75       0.000000        0.143403   \n",
       "1  0.004405   0.141491         1        149       0.004405        0.284895   \n",
       "2  0.039648   0.126195        10        215       0.044053        0.411090   \n",
       "3  0.061674   0.116635        24        276       0.105727        0.527725   \n",
       "4  0.088106   0.105163        44        331       0.193833        0.632887   \n",
       "5  0.079295   0.108987        62        388       0.273128        0.741874   \n",
       "6  0.127753   0.087954        91        434       0.400881        0.829828   \n",
       "7  0.162996   0.072658       128        472       0.563877        0.902486   \n",
       "8  0.198238   0.057361       173        502       0.762115        0.959847   \n",
       "9  0.237885   0.040153       227        523       1.000000        1.000000   \n",
       "\n",
       "         ks  \n",
       "0 -0.143403  \n",
       "1 -0.280490  \n",
       "2 -0.367037  \n",
       "3 -0.421998  \n",
       "4 -0.439055  \n",
       "5 -0.468746  \n",
       "6 -0.428947  \n",
       "7 -0.338609  \n",
       "8 -0.197732  \n",
       "9  0.000000  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tr_bucket = toad.metrics.KS_bucket(EYtr_proba,Ytr,bucket=10,method='quantile')\n",
    "tr_bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "----\n",
    "### V. Scorecard transformation\n",
    "Toad allows scorecard transformation from bins and convert probability of credit risk into scores accordingly.\n",
    "1. toad.scorecard.ScoreCard():\n",
    "\n",
    "1.1 ScoreCard().predict(X): output scores with original features. The function uses combiner and transer to bin the original feature to predict the risk scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/zhouxiyu/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>status.of.existing.checking.account</td>\n",
       "      <td>135.47</td>\n",
       "      <td>no checking account</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>status.of.existing.checking.account</td>\n",
       "      <td>93.30</td>\n",
       "      <td>... &gt;= 200 DM / salary assignments for at leas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>status.of.existing.checking.account</td>\n",
       "      <td>38.84</td>\n",
       "      <td>0 &lt;= ... &lt; 200 DM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>status.of.existing.checking.account</td>\n",
       "      <td>10.35</td>\n",
       "      <td>... &lt; 0 DM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>duration.in.month</td>\n",
       "      <td>137.15</td>\n",
       "      <td>[-inf ~ 9)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>duration.in.month</td>\n",
       "      <td>80.06</td>\n",
       "      <td>[9 ~ 18)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>duration.in.month</td>\n",
       "      <td>57.27</td>\n",
       "      <td>[18 ~ 33)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>duration.in.month</td>\n",
       "      <td>16.05</td>\n",
       "      <td>[33 ~ inf)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>credit.history</td>\n",
       "      <td>91.23</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>credit.history</td>\n",
       "      <td>61.13</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name   score  \\\n",
       "0  status.of.existing.checking.account  135.47   \n",
       "1  status.of.existing.checking.account   93.30   \n",
       "2  status.of.existing.checking.account   38.84   \n",
       "3  status.of.existing.checking.account   10.35   \n",
       "4                    duration.in.month  137.15   \n",
       "5                    duration.in.month   80.06   \n",
       "6                    duration.in.month   57.27   \n",
       "7                    duration.in.month   16.05   \n",
       "8                       credit.history   91.23   \n",
       "9                       credit.history   61.13   \n",
       "\n",
       "                                               value  \n",
       "0                                no checking account  \n",
       "1  ... >= 200 DM / salary assignments for at leas...  \n",
       "2                                  0 <= ... < 200 DM  \n",
       "3                                         ... < 0 DM  \n",
       "4                                         [-inf ~ 9)  \n",
       "5                                           [9 ~ 18)  \n",
       "6                                          [18 ~ 33)  \n",
       "7                                         [33 ~ inf)  \n",
       "8  critical account/ other credits existing (not ...  \n",
       "9           existing credits paid back duly till now  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 1: send combiner, transer, and LR parameters to scorecard (toad version >= 0.0.58)\n",
    "card = toad.scorecard.ScoreCard(combiner = combiner, transer = transer , C = 0.1) \n",
    "card.fit(Xtr, Ytr)\n",
    "card.export(to_frame = True,).head(10) # output the scorecard details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample scores: [645.78897702 395.25681998 620.57329911 542.42466191 529.69132256\n",
      " 566.04541878 583.90129633 474.56756095 550.71071912 609.11779198]\n",
      "Test KS:  0.4166860150143178\n"
     ]
    }
   ],
   "source": [
    "# Step 2: predict scores\n",
    "pred_scores = card.predict(data_ts)\n",
    "print('Sample scores:',pred_scores[:10])\n",
    "print('Test KS: ',KS(pred_scores, data_ts['creditability']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
